Oct 30 19:27:19.356552 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 19:27:19.356563 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 19:27:19.356570 ip-10-0-139-103 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 19:27:19.356808 ip-10-0-139-103 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 19:27:29.604203 ip-10-0-139-103 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 19:27:29.604218 ip-10-0-139-103 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0f5b541f697e49e8ab76f8fc80da20c0 -- Oct 30 19:29:26.452720 ip-10-0-139-103 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:29:26.829320 ip-10-0-139-103 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:29:26.829320 ip-10-0-139-103 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:29:26.829320 ip-10-0-139-103 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:29:26.829320 ip-10-0-139-103 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:29:26.833315 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833233 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.833315 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833315 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833320 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833325 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833329 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833333 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833336 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833339 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833342 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833344 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833347 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833350 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833353 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833355 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833358 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833361 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833363 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833369 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833372 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833375 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.833387 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833377 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833380 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833382 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833385 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833388 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833391 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833394 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833396 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833400 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833404 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833407 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833410 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833413 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833416 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833419 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833422 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833424 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833446 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833449 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.833840 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833452 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833454 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833457 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833459 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833462 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833465 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833467 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833470 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833472 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833474 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833483 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833485 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833488 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833491 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833493 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833497 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833500 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833502 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833505 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833507 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.834282 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833510 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833512 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833515 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833517 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833520 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833522 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833525 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833527 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833530 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833532 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833535 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833537 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833540 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833542 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833545 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833547 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833550 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833552 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833560 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833563 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.834784 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833566 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833568 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833571 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833574 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833577 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833579 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833582 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833585 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833588 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833590 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833593 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833595 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833598 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833600 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833603 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833605 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833608 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833611 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833613 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.835379 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833616 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833618 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833621 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833624 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833626 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833628 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833632 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.833634 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834487 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834494 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834499 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834502 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834506 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834508 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834511 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834514 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834516 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834525 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834527 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.836100 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834530 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834533 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834536 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834538 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834541 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834544 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834546 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834549 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834551 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834554 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834556 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834558 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834561 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834563 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834566 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834568 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834571 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834573 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834576 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834578 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.836626 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834581 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834583 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834587 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834590 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834593 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834595 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834598 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834600 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834603 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834605 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834608 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834617 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834620 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834623 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834626 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834628 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834631 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834634 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834636 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834638 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.837106 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834641 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834643 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834646 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834648 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834651 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834653 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834655 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834658 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834661 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834663 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834666 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834668 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834671 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834673 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834676 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834679 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834682 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834684 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834687 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834689 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.837600 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834692 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834695 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834697 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834700 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834708 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834711 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834714 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834716 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834719 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834722 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834724 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834727 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834729 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834733 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834735 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834738 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834740 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834743 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834746 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.838102 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834748 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834751 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834753 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834755 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834758 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834762 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834765 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834767 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834770 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834773 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834777 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834780 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834783 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834788 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834792 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.834795 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836194 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836207 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836211 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.838573 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836214 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836217 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836220 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836224 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836229 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836232 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836234 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836237 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836240 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836242 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836245 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836248 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836250 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836253 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836256 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836258 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836261 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836263 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836267 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.839109 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836270 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836272 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836275 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836278 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836281 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836283 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836286 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836289 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836291 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836294 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836297 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836299 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836302 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836304 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836307 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836309 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836312 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836315 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836317 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.839569 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836320 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836322 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836325 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836327 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836330 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836333 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836335 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836340 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836342 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836345 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836348 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836350 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836353 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836355 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836358 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836360 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836363 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836366 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836369 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.840024 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836372 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836375 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836378 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836380 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836383 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836385 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836388 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836392 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836395 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836398 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836401 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836404 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836406 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836409 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836411 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836413 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836416 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836418 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836421 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836423 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.840489 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836426 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836442 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836447 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836451 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836454 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836457 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836459 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836462 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836464 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836467 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836470 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836472 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836475 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836478 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836481 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836483 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836486 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836490 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836493 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836495 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.840970 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836498 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836501 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836503 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836506 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836509 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.836511 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837472 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837482 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837488 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837494 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837498 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837501 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837506 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837510 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837514 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837517 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837521 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837525 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837528 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837531 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837534 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837537 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837541 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:29:26.841464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837544 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837549 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837553 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837556 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837559 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837562 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837566 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837570 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837573 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837576 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837579 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837582 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837585 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837589 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837591 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837596 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837598 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837601 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837604 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837608 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837611 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837616 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837619 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837622 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:29:26.842002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837625 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837629 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837634 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837637 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837641 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837644 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837647 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837650 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837653 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837656 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837659 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837662 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837664 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837669 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837672 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837675 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837678 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837681 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837685 2583 flags.go:64] FLAG: --help="false" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837688 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837691 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837694 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837697 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:29:26.842643 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837701 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837704 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837707 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837710 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837713 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837716 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837719 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837724 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837727 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837730 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837733 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837737 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837741 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837744 2583 flags.go:64] FLAG: --lock-file="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837748 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837751 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837754 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837760 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837763 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837766 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837768 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837772 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837775 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837778 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837781 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:29:26.843195 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837785 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837788 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837792 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837795 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837798 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837801 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837804 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837807 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837810 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837813 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837821 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837824 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837827 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837831 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837835 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837838 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837841 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837844 2583 flags.go:64] FLAG: --port="10250" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837847 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837851 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-012b7be2a08857dfd" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837854 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837857 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837860 2583 flags.go:64] FLAG: --register-node="true" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837863 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837870 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837873 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:29:26.843857 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837876 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837879 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837883 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837886 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837889 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837892 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837895 2583 flags.go:64] FLAG: --runonce="false" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837898 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837901 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837904 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837906 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837910 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837913 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837916 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837919 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837922 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837925 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837928 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837931 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837934 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837938 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837941 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837947 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837952 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837955 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:29:26.844533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837959 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837962 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837965 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837968 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837971 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837974 2583 flags.go:64] FLAG: --v="2" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837979 2583 flags.go:64] FLAG: --version="false" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837984 2583 flags.go:64] FLAG: --vmodule="" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837988 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.837991 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838123 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838126 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838129 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838131 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838134 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838137 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838139 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838142 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838144 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838147 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838150 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838153 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.845118 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838155 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838158 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838160 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838163 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838165 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838170 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838172 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838175 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838179 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838182 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838186 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838190 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838193 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838196 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838199 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838202 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838204 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838207 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838210 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.845654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838212 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838215 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838217 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838220 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838222 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838225 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838227 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838230 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838232 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838235 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838238 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838240 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838243 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838245 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838248 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838251 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838253 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838255 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838259 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838261 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.846103 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838265 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838269 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838272 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838274 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838277 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838279 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838282 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838285 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838288 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838291 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838293 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838296 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838298 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838301 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838304 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838306 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838309 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838312 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838314 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.846612 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838317 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838319 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838323 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838327 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838330 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838333 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838336 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838338 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838341 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838343 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838346 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838349 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838352 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838354 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838360 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838362 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838365 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838367 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838370 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.847095 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838372 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838375 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838378 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838381 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838383 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838386 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838389 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838392 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838394 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838397 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838399 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838402 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838404 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838407 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838409 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838412 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.847618 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838415 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.838455 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838500 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838505 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838508 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838511 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838514 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838516 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838521 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838524 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838527 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838529 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838533 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838536 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838538 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838541 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.848003 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838543 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838546 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838548 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838552 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838555 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838557 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838560 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838563 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838565 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838568 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838570 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838573 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838576 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838578 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838581 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838583 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838586 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838589 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838592 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838595 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.848383 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838597 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838599 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838602 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838605 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838609 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838613 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838616 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838619 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838622 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838625 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838627 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838630 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838632 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838634 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838637 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838639 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838643 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838645 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838647 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.848864 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838650 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838652 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838655 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838657 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838660 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838663 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838665 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838668 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838670 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838673 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838676 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838679 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838681 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838684 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838686 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838689 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838692 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838694 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838697 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838699 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.849336 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838702 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838704 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838707 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838709 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838711 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838714 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838716 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838719 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838721 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838727 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838730 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838733 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838735 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838737 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838740 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838743 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838746 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838748 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.849844 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838751 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838753 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838757 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838761 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838763 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838766 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838768 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838772 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838775 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838777 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838780 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838782 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838785 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.838787 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.845629 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.845643 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.845660 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.845665 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845722 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.850298 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845728 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845731 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845736 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845741 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845745 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845748 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845751 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845754 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845757 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845759 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845762 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845764 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845767 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845770 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845773 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845776 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845778 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845781 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845784 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.850778 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845787 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845789 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845792 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845795 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845797 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845800 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845802 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845805 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845809 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845812 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845814 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845817 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845819 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845822 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845825 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845827 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845830 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845832 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845836 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.851228 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845838 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845841 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845843 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845846 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845848 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845851 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845853 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845856 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845859 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845861 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845864 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845866 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845869 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845872 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845874 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845877 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845879 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845882 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845884 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845887 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.851743 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845889 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845894 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845897 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845899 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845902 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845905 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845908 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845910 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845913 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845915 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845918 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845920 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845927 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845930 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845933 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845935 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845938 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845941 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845943 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845946 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.852275 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845948 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845951 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845954 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845956 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845959 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845962 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845964 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845967 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845969 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845972 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845975 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845977 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845981 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845986 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845989 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845991 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845994 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845997 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.845999 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846002 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.852759 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846004 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846007 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846009 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846012 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846014 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846018 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846020 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.846048 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846189 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846194 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846197 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846200 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846203 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846206 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846208 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.853299 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846211 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846213 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846216 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846219 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846222 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846224 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846226 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846229 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846232 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846234 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846237 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846239 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846242 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846245 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846248 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846250 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846253 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846256 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846260 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.853671 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846262 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846265 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846267 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846270 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846274 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846276 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846279 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846282 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846284 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846286 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846289 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846292 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846294 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846297 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846299 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846302 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846304 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846307 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846310 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.854141 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846312 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846315 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846317 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846320 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846323 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846325 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846328 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846330 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846333 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846336 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846338 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846341 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846343 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846346 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846348 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846351 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846353 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846356 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846360 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846362 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.854595 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846365 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846367 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846370 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846372 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846375 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846377 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846380 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846382 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846385 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846387 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846390 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846393 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846397 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846401 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846404 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846407 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846409 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846412 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846414 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846417 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.855065 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846419 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846422 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846424 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846427 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846444 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846448 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846450 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846453 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846455 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846458 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846460 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846464 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846466 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846469 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846472 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846475 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846477 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846480 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846482 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.855532 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846485 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846488 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.846511 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846548 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846552 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846555 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846558 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846560 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846563 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846565 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846568 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846571 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846573 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846576 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846578 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846581 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846583 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:29:26.855977 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846586 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846588 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846590 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846593 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846598 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846602 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846605 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846607 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846612 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846615 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846618 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846621 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846623 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846626 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846629 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846631 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846634 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846636 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846639 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:29:26.856375 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846642 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846644 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846647 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846649 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846652 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846654 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846657 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846660 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846663 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846665 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846668 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846671 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846673 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846676 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846678 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846681 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846683 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846686 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846688 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846691 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:29:26.856842 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846693 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846698 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846701 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846703 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846706 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846708 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846713 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846715 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846718 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846720 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846723 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846726 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846728 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846731 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846734 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846736 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846738 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846741 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846746 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:29:26.857369 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846749 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846752 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846756 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846759 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846762 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846765 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846768 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846771 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846774 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846776 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846779 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846781 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846784 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846786 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846791 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846793 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846796 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846799 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846801 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:29:26.857923 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846806 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846808 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846811 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846813 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846816 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846819 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846821 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846824 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846827 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846830 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846833 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846836 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846838 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846841 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:26.846843 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.846893 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.849175 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.850065 2583 server.go:1014] "Starting client certificate rotation" Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.850166 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:29:26.858716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.850201 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:29:26.859217 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.851951 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:29:26.859217 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.853047 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:29:26.861978 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.861959 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:29:26.865770 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.865753 2583 log.go:25] "Validated CRI v1 image API" Oct 30 19:29:26.866595 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.866570 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:29:26.872210 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.872182 2583 fs.go:135] Filesystem UUIDs: map[25672f9a-2eab-4922-8e2b-2b8cd1ac713f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e31d558f-d365-425b-8523-a1faabfb0c51:/dev/nvme0n1p3] Oct 30 19:29:26.872210 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.872201 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:29:26.878339 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.878229 2583 manager.go:217] Machine: {Timestamp:2026-10-30 19:29:26.876900345 +0000 UTC m=+0.300856523 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097357 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9e0d17a42c5d1f72965d2b035f7d SystemUUID:ec2a9e0d-17a4-2c5d-1f72-965d2b035f7d BootID:0f5b541f-697e-49e8-ab76-f8fc80da20c0 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e6:ca:a0:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e6:ca:a0:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:4b:03:3c:49:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:29:26.878937 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.878926 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:29:26.879018 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.879006 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:29:26.881768 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881744 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:29:26.881897 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881767 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-103.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:29:26.881951 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881899 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:29:26.881951 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881906 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:29:26.882006 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881953 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:29:26.882006 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.881971 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:29:26.883110 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.883099 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:29:26.887655 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887640 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:29:26.887794 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887784 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:29:26.887839 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887819 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:29:26.887839 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887829 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:29:26.887839 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887839 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:29:26.887963 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887858 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:29:26.887963 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.887866 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:29:26.888205 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.888193 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:29:26.888680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.888660 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:29:26.890455 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.890421 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:29:26.896404 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.896387 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:29:26.897051 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897038 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:29:26.897120 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897068 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:29:26.897230 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897220 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:29:26.897280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897236 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:29:26.897280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897243 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:29:26.897280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897250 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:29:26.897280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897260 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:29:26.897280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897269 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:29:26.897807 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897795 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:29:26.897807 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897805 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:29:26.897888 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897813 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:29:26.897888 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897819 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:29:26.897888 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897829 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:29:26.897888 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.897838 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:29:26.899115 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.899104 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:29:26.899159 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.899117 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:29:26.902758 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.902745 2583 server.go:1263] "Started kubelet" Oct 30 19:29:26.902847 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.902817 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:29:26.902909 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.902856 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:29:26.902945 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.902928 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:29:26.903508 ip-10-0-139-103 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:29:26.904191 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.904173 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:29:26.906490 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.906469 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-103.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:29:26.909176 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.909164 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:29:26.910210 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.910191 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:29:26.912791 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.912763 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xkpgw" Oct 30 19:29:26.913034 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.913018 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:29:26.913550 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.913535 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.913640 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914005 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914032 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914006 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914094 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:29:26.914144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914102 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:29:26.914426 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914395 2583 factory.go:55] Registering systemd factory Oct 30 19:29:26.914426 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914410 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:29:26.914646 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914636 2583 factory.go:153] Registering CRI-O factory Oct 30 19:29:26.914684 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914649 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 19:29:26.914711 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914694 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:29:26.914743 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914713 2583 factory.go:103] Registering Raw factory Oct 30 19:29:26.914743 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.914727 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 19:29:26.915573 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.915559 2583 manager.go:319] Starting recovery of all containers Oct 30 19:29:26.924494 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.924458 2583 manager.go:324] Recovery completed Oct 30 19:29:26.924973 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.924953 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xkpgw" Oct 30 19:29:26.925514 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.925488 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:29:26.925514 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.925490 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-103.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:29:26.925636 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.925609 2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-103.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:29:26.925929 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.924924 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-103.us-east-2.compute.internal.18e365665a7bb153 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-103.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-103.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:29:26.902714707 +0000 UTC m=+0.326670884,LastTimestamp:2026-10-30 19:29:26.902714707 +0000 UTC m=+0.326670884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-103.us-east-2.compute.internal,}" Oct 30 19:29:26.927313 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.927291 2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:29:26.928849 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.928835 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:26.930526 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.930505 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:26.930604 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.930544 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:26.930604 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.930560 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:26.931152 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931133 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:29:26.931152 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931149 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:29:26.931291 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931168 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:26.931291 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931189 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:29:26.931921 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931905 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:26.931993 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931931 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:26.931993 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.931942 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:26.933192 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.933178 2583 policy_none.go:50] "Start" Oct 30 19:29:26.933245 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.933199 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:29:26.933245 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.933213 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:29:26.936386 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.936375 2583 policy_none.go:44] "Start" Oct 30 19:29:26.988017 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.987988 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:29:26.991829 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.991812 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 19:29:26.991928 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.991853 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:29:26.991928 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.991865 2583 server.go:87] "Starting device plugin registration server" Oct 30 19:29:26.992121 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.992106 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:29:26.992209 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.992122 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:29:26.992271 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.992259 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:29:26.992350 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.992322 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:29:26.992350 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:26.992334 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:29:26.992849 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.992833 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:29:26.992912 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:26.992877 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.015235 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.015220 2583 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2616 into the system container \"/system.slice\": " Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.092539 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.093675 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.093701 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.093712 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.093744 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.096782 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.098058 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.098077 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.098093 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:29:27.099549 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.098123 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:29:27.101967 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.101289 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:29:27.102154 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.102139 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.102199 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.102160 2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-103.us-east-2.compute.internal\": node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.144918 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.144893 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.199010 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.198988 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal"] Oct 30 19:29:27.199071 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.199042 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:27.199992 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.199968 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:27.200050 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.199997 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:27.200050 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.200006 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:27.202246 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.202234 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:27.202383 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.202369 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.202449 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.202398 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:27.203006 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.202992 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:27.203071 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.203012 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:27.203071 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.203021 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:27.203071 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.203044 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:27.203071 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.203065 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:27.203228 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.203078 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:27.205232 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.205218 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.205289 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.205241 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:29:27.205916 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.205900 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:29:27.206012 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.205926 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:29:27.206012 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.205936 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:29:27.215923 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.215907 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.216015 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.215931 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.216015 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.215948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.223863 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.223835 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.225817 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.225803 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.245616 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.245598 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.316518 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316495 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.316598 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316523 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.316598 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.316598 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316590 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.316692 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316594 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.316692 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.316603 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.346172 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.346151 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.446965 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.446890 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.528356 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.528336 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.530512 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.530497 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:27.547827 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.547798 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.648329 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.648301 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.748824 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.748771 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.849313 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:27.849287 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:29:27.850389 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.850371 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:29:27.850527 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:27.850511 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:29:29.184469 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.184451 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:29:29.192545 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.192502 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:28 +0000 UTC" deadline="2028-08-09 07:00:06.469345036 +0000 UTC" Oct 30 19:29:29.192545 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.192541 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15563h30m37.276806519s" Oct 30 19:29:29.276279 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.276226 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:29.290574 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.290554 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:29:29.292964 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.292950 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:29:29.303701 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.303683 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:29:29.427072 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:29.427010 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4925bc4d897d213bab4d8b38932653e.slice/crio-fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85 WatchSource:0}: Error finding container fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85: Status 404 returned error can't find the container with id fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85 Oct 30 19:29:29.427585 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:29.427562 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf224810f96f29df31a3e8747164653d1.slice/crio-2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81 WatchSource:0}: Error finding container 2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81: Status 404 returned error can't find the container with id 2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81 Oct 30 19:29:29.432028 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.432013 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:29:29.668539 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.668460 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:29:29.683042 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:29.683011 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:29:30.150408 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.150332 2583 apiserver.go:52] "Watching apiserver" Oct 30 19:29:30.162749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.162722 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:29:30.163124 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.163097 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-kms8d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn","openshift-cluster-node-tuning-operator/tuned-2qlwk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal","openshift-dns/node-resolver-jcmd8","openshift-image-registry/node-ca-7gh2g","openshift-multus/multus-2hcjh","openshift-multus/network-metrics-daemon-8nnkd","kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-l89fd","openshift-network-diagnostics/network-check-target-mrpl8","openshift-network-operator/iptables-alerter-2pmgc","openshift-ovn-kubernetes/ovnkube-node-h6kbw","kube-system/global-pull-secret-syncer-jllfr"] Oct 30 19:29:30.165608 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.165584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.167899 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.167879 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.170433 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.170185 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.171322 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.171292 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.171420 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.171379 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.171484 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.171471 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:29:30.171572 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.171561 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:29:30.172472 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.172446 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.172590 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.172528 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:30.173134 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173112 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\"" Oct 30 19:29:30.173243 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173166 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:29:30.173464 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173448 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.173523 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173493 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:29:30.173571 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173558 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\"" Oct 30 19:29:30.173660 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173647 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:29:30.173702 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173690 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.173785 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.173773 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:29:30.175165 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175017 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:29:30.175165 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175105 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.175770 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175749 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:29:30.175877 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175798 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.175877 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175821 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.175877 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.175873 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\"" Oct 30 19:29:30.177019 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.176977 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.179816 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.179391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.179816 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.179462 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:30.181474 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.181446 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\"" Oct 30 19:29:30.182782 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.182762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.184966 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.184943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185068 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.184983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185068 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185174 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185174 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185138 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.185174 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.185673 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.185673 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185664 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185690 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185712 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.185805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185759 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.185805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.185988 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185838 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.185988 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185866 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.185988 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185988 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.185988 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185974 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.185992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575rj\" (UniqueName: \"kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186006 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186036 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6s9s\" (UniqueName: \"kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186161 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186181 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff89l\" (UniqueName: \"kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186251 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186265 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.186297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186304 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186435 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186480 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186557 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186611 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186664 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqhrb\" (UniqueName: \"kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.186725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186763 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186803 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lvf\" (UniqueName: \"kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.187227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.186834 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.190344 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.190324 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.191078 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.190916 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.193597 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.193573 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:30.193703 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.193653 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:30.195855 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.195831 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.196687 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.196631 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:29:30.210256 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.210238 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhk4q" Oct 30 19:29:30.222528 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.222506 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lhk4q" Oct 30 19:29:30.271877 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.271852 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:29:30.271997 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.271865 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:29:30.271997 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.271865 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.271997 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.271933 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.271997 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.271966 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.273721 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.273702 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.273721 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.273718 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:29:30.274070 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.274038 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.274167 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.274147 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:29:30.274266 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.274227 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.274266 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.274228 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:29:30.275497 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.275478 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:29:30.275592 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.275482 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\"" Oct 30 19:29:30.276297 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276271 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:29:30.276381 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276344 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\"" Oct 30 19:29:30.276450 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276387 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\"" Oct 30 19:29:30.276502 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276484 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\"" Oct 30 19:29:30.276559 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276542 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:29:30.276613 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.276596 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\"" Oct 30 19:29:30.277445 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.277428 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:29:30.287959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287801 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.287959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.287959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287874 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.287959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.287959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288248 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.287981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288248 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288028 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.288248 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288135 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288370 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288351 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.288473 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288504 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.288532 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28rx\" (UniqueName: \"kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.288563 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288545 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288594 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288569 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288627 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288593 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.288627 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288617 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.288687 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288637 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.288687 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288657 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288687 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288673 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288715 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288752 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288766 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.288788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288780 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288813 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhrb\" (UniqueName: \"kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288910 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288929 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lvf\" (UniqueName: \"kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288957 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.288970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288971 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.288985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289000 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289013 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289070 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289107 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289132 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7xx\" (UniqueName: \"kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289155 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289201 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289250 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289273 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575rj\" (UniqueName: \"kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289297 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289321 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:30.289530 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289350 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289382 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff89l\" (UniqueName: \"kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289413 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289479 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289583 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289630 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289654 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289679 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g982p\" (UniqueName: \"kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289711 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289736 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289760 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289783 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.290136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289833 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289891 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9m54\" (UniqueName: \"kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289968 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.289996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290053 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290093 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290144 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s9s\" (UniqueName: \"kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.290776 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290577 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.291284 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.290993 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.291284 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.291205 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.291284 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.291248 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.291578 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.291557 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:29:30.291844 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.291824 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.291930 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.291916 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.292121 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292101 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.292189 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292171 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.292290 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292272 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.292341 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.292384 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292367 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.292430 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292407 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.292498 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292480 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.292637 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292623 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.292722 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.292709 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.293123 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293106 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.293199 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293172 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.293252 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293220 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.293301 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293268 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.293503 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293486 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.293588 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293531 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.293648 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293610 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.293700 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.293672 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.293750 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.293696 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:30.293797 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.293758 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:29:30.793738167 +0000 UTC m=+2.956068938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:30.294140 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294116 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.294224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294146 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.294224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294198 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.294224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294212 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.294370 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294243 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.294370 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294241 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.294370 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294300 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.294370 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294309 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.294564 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294371 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.294564 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294415 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.294564 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294441 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.294564 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294459 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.295088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294786 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.295088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.294922 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.295261 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.295121 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.296577 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.296555 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.296805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.296789 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.304409 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.303943 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s9s\" (UniqueName: \"kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.305342 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.305321 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhrb\" (UniqueName: \"kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.305855 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.305833 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff89l\" (UniqueName: \"kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.307038 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.307017 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lvf\" (UniqueName: \"kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.307734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.307716 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-575rj\" (UniqueName: \"kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.338317 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.338285 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:29:30.364573 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.364525 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerStarted","Data":"2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81"} Oct 30 19:29:30.365740 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.365705 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"e4925bc4d897d213bab4d8b38932653e","Type":"ContainerStarted","Data":"fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85"} Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391185 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391219 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391244 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g982p\" (UniqueName: \"kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391269 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391315 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m54\" (UniqueName: \"kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391394 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391417 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391439 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391485 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rx\" (UniqueName: \"kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391506 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391527 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.395956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391646 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391669 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391704 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391778 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7xx\" (UniqueName: \"kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391843 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391897 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.391944 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392026 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392175 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.396795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392280 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392512 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392554 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392588 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.392777 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.393354 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.393422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.393802 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.393839 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394381 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394443 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394482 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394517 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394564 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394610 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394664 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394772 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.397575 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394806 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.398391 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.394842 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.398391 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.395244 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:30.398391 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.395311 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:30.895293172 +0000 UTC m=+3.057623931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:30.398391 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.395355 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.398602 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.398539 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.399661 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.399637 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.407426 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.406846 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:30.407426 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.406886 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:30.407426 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.406900 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:30.407426 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.406965 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:30.906948458 +0000 UTC m=+3.069279235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:30.408259 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.408235 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28rx\" (UniqueName: \"kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.408479 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.408452 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m54\" (UniqueName: \"kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.409547 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.409523 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7xx\" (UniqueName: \"kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.409908 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.409886 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g982p\" (UniqueName: \"kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.494585 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.494551 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:29:30.504647 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.504617 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651c14dc_17ea_4608_b927_ba1af3ea8b41.slice/crio-71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755 WatchSource:0}: Error finding container 71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755: Status 404 returned error can't find the container with id 71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755 Oct 30 19:29:30.507179 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.507154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:29:30.515236 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.515146 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:29:30.516026 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.516006 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:30.516469 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.516409 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b76e60_de88_4c65_b570_fb4772dfbb5b.slice/crio-c4ac8f7d07c54c8e55536bdd75eca48bff7d1c40705fccc3ed6815dc4ee99aca WatchSource:0}: Error finding container c4ac8f7d07c54c8e55536bdd75eca48bff7d1c40705fccc3ed6815dc4ee99aca: Status 404 returned error can't find the container with id c4ac8f7d07c54c8e55536bdd75eca48bff7d1c40705fccc3ed6815dc4ee99aca Oct 30 19:29:30.517874 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.517845 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:29:30.525673 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.525641 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2hcjh" Oct 30 19:29:30.533741 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.532494 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffaa0f5e_c8f0_4e96_83a0_61a39edd65ce.slice/crio-e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a WatchSource:0}: Error finding container e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a: Status 404 returned error can't find the container with id e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a Oct 30 19:29:30.535442 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.535412 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36aff52_af76_477f_990d_9900e4c43954.slice/crio-305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7 WatchSource:0}: Error finding container 305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7: Status 404 returned error can't find the container with id 305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7 Oct 30 19:29:30.537390 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.537365 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed84a66_cd93_4e58_adb3_e097e1b17a14.slice/crio-bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e WatchSource:0}: Error finding container bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e: Status 404 returned error can't find the container with id bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e Oct 30 19:29:30.538592 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.538087 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:29:30.541714 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.541687 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:30.548675 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.548501 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:29:30.551654 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.551635 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8efd1a0_6c9c_4f9a_bbb8_2a280f2b94d5.slice/crio-8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813 WatchSource:0}: Error finding container 8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813: Status 404 returned error can't find the container with id 8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813 Oct 30 19:29:30.553669 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.553642 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46dad1ef_96d8_483e_909b_c082e47acbbd.slice/crio-02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6 WatchSource:0}: Error finding container 02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6: Status 404 returned error can't find the container with id 02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6 Oct 30 19:29:30.561076 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:29:30.561038 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29bace5a_fb35_45e9_a2ce_41a61af06524.slice/crio-467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63 WatchSource:0}: Error finding container 467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63: Status 404 returned error can't find the container with id 467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63 Oct 30 19:29:30.794732 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.794636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:30.794893 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.794844 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:30.794958 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.794944 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:29:31.794921691 +0000 UTC m=+3.957252468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:30.895556 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.895521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:30.895727 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.895697 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:30.895801 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.895778 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:31.895758045 +0000 UTC m=+4.058088802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:30.996660 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:30.996624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:30.996847 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.996770 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:30.996847 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.996786 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:30.996847 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.996796 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:30.996847 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:30.996844 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:31.996828555 +0000 UTC m=+4.159159323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:31.226568 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.225990 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:30 +0000 UTC" deadline="2028-08-08 16:35:14.564282854 +0000 UTC" Oct 30 19:29:31.226568 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.226052 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15549h5m43.338233767s" Oct 30 19:29:31.361444 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.360458 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:31.361444 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:31.360865 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:31.383591 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.383552 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"e4925bc4d897d213bab4d8b38932653e","Type":"ContainerStarted","Data":"f42d28204a14dc0a820e7be2809b8e8a9eb02a6b0a9612a35b861724fd22699f"} Oct 30 19:29:31.386248 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.386219 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pmgc" event={"ID":"29bace5a-fb35-45e9-a2ce-41a61af06524","Type":"ContainerStarted","Data":"467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63"} Oct 30 19:29:31.396348 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.396303 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kms8d" event={"ID":"46dad1ef-96d8-483e-909b-c082e47acbbd","Type":"ContainerStarted","Data":"02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6"} Oct 30 19:29:31.399093 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.399067 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gh2g" event={"ID":"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce","Type":"ContainerStarted","Data":"e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a"} Oct 30 19:29:31.404756 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.404656 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7"} Oct 30 19:29:31.406087 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.405708 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" podStartSLOduration=2.405693446 podStartE2EDuration="2.405693446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:31.40534845 +0000 UTC m=+3.567679228" watchObservedRunningTime="2026-10-30 19:29:31.405693446 +0000 UTC m=+3.568024223" Oct 30 19:29:31.408363 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.408340 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" event={"ID":"e8b76e60-de88-4c65-b570-fb4772dfbb5b","Type":"ContainerStarted","Data":"c4ac8f7d07c54c8e55536bdd75eca48bff7d1c40705fccc3ed6815dc4ee99aca"} Oct 30 19:29:31.409795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.409771 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813"} Oct 30 19:29:31.411283 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.411258 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hcjh" event={"ID":"eed84a66-cd93-4e58-adb3-e097e1b17a14","Type":"ContainerStarted","Data":"bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e"} Oct 30 19:29:31.413628 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.413605 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jcmd8" event={"ID":"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f","Type":"ContainerStarted","Data":"af7a1cc691ca4595e6bf1f7f8cd9918f3baab302f5b91276c2a5455c5c883883"} Oct 30 19:29:31.415623 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.415599 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerStarted","Data":"71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755"} Oct 30 19:29:31.801340 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.801305 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:31.801591 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:31.801484 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:31.801591 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:31.801547 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:29:33.801529669 +0000 UTC m=+5.963860440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:31.902968 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:31.902292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:31.902968 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:31.902421 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:31.902968 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:31.902480 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:33.902461542 +0000 UTC m=+6.064792299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:32.002796 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.002759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:32.002977 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.002942 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:32.002977 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.002961 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:32.002977 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.002973 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:32.003162 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.003029 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:34.003010975 +0000 UTC m=+6.165341743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:32.227234 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.227103 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:30 +0000 UTC" deadline="2028-04-08 16:20:37.109440837 +0000 UTC" Oct 30 19:29:32.227234 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.227141 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12620h51m4.882304171s" Oct 30 19:29:32.363720 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.360810 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:32.363720 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.360934 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:32.363720 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.361073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:32.363720 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:32.361210 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:32.427139 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.427099 2583 generic.go:356] "Generic (PLEG): container finished" podID="f224810f96f29df31a3e8747164653d1" containerID="680303dc2b2394ba09904382e111d2b816b6236954b8c0b2fa5272d3e4e32c1e" exitCode=0 Oct 30 19:29:32.427360 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:32.427251 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerDied","Data":"680303dc2b2394ba09904382e111d2b816b6236954b8c0b2fa5272d3e4e32c1e"} Oct 30 19:29:33.361092 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.360896 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:33.361092 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:33.361030 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:33.455506 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.455469 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="cbfa5d3ba915e9089abeffaeef805886838dc7d5060a2e7395a594137392978d" exitCode=0 Oct 30 19:29:33.455686 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.455588 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"cbfa5d3ba915e9089abeffaeef805886838dc7d5060a2e7395a594137392978d"} Oct 30 19:29:33.466204 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.466178 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerStarted","Data":"5db72b8e3365cd2a84d9df19edbca5a4d2bb74d11260134d6f502e3457d04dec"} Oct 30 19:29:33.509046 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.508970 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" podStartSLOduration=4.508925834 podStartE2EDuration="4.508925834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:33.50796897 +0000 UTC m=+5.670299747" watchObservedRunningTime="2026-10-30 19:29:33.508925834 +0000 UTC m=+5.671256609" Oct 30 19:29:33.817410 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.817321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:33.817553 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:33.817512 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:33.817614 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:33.817563 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:29:37.817550496 +0000 UTC m=+9.979881251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:33.918085 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:33.918041 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:33.918246 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:33.918213 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:33.918307 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:33.918277 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:37.918262583 +0000 UTC m=+10.080593337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:34.019290 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:34.019255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:34.019471 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.019414 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:34.019471 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.019434 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:34.019471 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.019448 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:34.019617 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.019505 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:38.019488006 +0000 UTC m=+10.181818766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:34.360865 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:34.360829 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:34.361038 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.360987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:34.363842 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:34.363815 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:34.364273 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:34.363938 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:35.360405 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:35.360375 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:35.360618 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:35.360480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:35.484503 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:35.483940 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kms8d" event={"ID":"46dad1ef-96d8-483e-909b-c082e47acbbd","Type":"ContainerStarted","Data":"9e7003f06b1ea5205c106874d89b156a76a7cb8edcf241554462f565d4d0c3a4"} Oct 30 19:29:35.504313 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:35.504262 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kms8d" podStartSLOduration=3.58423579 podStartE2EDuration="7.504247847s" totalImagesPullingTime="3.920012057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:35.50411696 +0000 UTC m=+7.666447737" watchObservedRunningTime="2026-10-30 19:29:35.504247847 +0000 UTC m=+7.666578606" Oct 30 19:29:36.360572 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:36.360543 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:36.360718 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:36.360691 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:36.360966 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:36.360936 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:36.361072 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:36.361035 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:36.491225 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:36.491189 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7gh2g" event={"ID":"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce","Type":"ContainerStarted","Data":"49a9a7ad4a50d8d326a9a4474d1f0ef52da0780c710b1f1dc52b5872dfe26537"} Oct 30 19:29:37.142359 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.142328 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:37.142847 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.142800 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:37.162728 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.162674 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7gh2g" podStartSLOduration=4.273744396 podStartE2EDuration="9.162657638s" totalImagesPullingTime="4.888913242s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:36.513642386 +0000 UTC m=+8.675973162" watchObservedRunningTime="2026-10-30 19:29:37.162657638 +0000 UTC m=+9.324988415" Oct 30 19:29:37.360625 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.360540 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:37.360774 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:37.360677 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:37.414784 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.414751 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:37.415399 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.415377 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:29:37.498341 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.497868 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"6e17da26fcad1398b2c5f91ba8739150e073bf7233d06fb2c1ca47d7dcbecbff"} Oct 30 19:29:37.850803 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.850726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:37.850965 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:37.850861 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:37.850965 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:37.850926 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:29:45.850909212 +0000 UTC m=+18.013239980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:37.952373 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:37.951784 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:37.952373 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:37.951924 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:37.952373 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:37.952002 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:45.951981122 +0000 UTC m=+18.114311882 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:38.053088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.052529 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:38.053088 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.052697 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:38.053088 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.052718 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:38.053088 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.052732 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:38.053088 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.052788 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:29:46.052768234 +0000 UTC m=+18.215099009 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:38.173874 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.173840 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:29:38.261442 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.261340 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:29:38.173870744Z","UUID":"20b4fdae-f0a7-4b3b-a6ca-4c345ba248b9","Handler":null,"Name":"","Endpoint":""} Oct 30 19:29:38.266083 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.265038 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:29:38.266083 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.265089 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:29:38.363154 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.363041 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:38.363332 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.363291 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:38.363409 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.363337 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:38.363479 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:38.363448 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:38.503201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.503163 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"87d043921fbc201018fa8815a227531edc0fb6e7d424d11c84b9b8016baef344"} Oct 30 19:29:38.505796 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.505769 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2hcjh" event={"ID":"eed84a66-cd93-4e58-adb3-e097e1b17a14","Type":"ContainerStarted","Data":"4926b498f9cf3ce7ed0f583e3b30ebc57008412e424cf55915782d874f9aa77f"} Oct 30 19:29:38.509117 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.509082 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="d5eef995edce3684d810de402538ed3d5a3a237f158c291ea4d832418dd9e451" exitCode=0 Oct 30 19:29:38.509947 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.509924 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"d5eef995edce3684d810de402538ed3d5a3a237f158c291ea4d832418dd9e451"} Oct 30 19:29:38.558377 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:38.558133 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2hcjh" podStartSLOduration=3.178485114 podStartE2EDuration="10.558116072s" totalImagesPullingTime="7.379630958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:38.530180783 +0000 UTC m=+10.692511559" watchObservedRunningTime="2026-10-30 19:29:38.558116072 +0000 UTC m=+10.720446849" Oct 30 19:29:39.360856 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.360828 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:39.360987 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:39.360964 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:39.514240 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.514201 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"7b179ee08342fe252ca7c04d97c5e12b3f212a91247e4b634ee33b1d94cc0a29"} Oct 30 19:29:39.516570 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.516534 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jcmd8" event={"ID":"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f","Type":"ContainerStarted","Data":"a22dc647a38f4f82147bf82d50ab90afb2423d41fa4e0a8097d4352ef28e9999"} Oct 30 19:29:39.520088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.519959 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="30739ea3520f26c0b57c7d495296a68e2e038b1d212036aeb6424ef366d1006f" exitCode=0 Oct 30 19:29:39.520088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.520017 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"30739ea3520f26c0b57c7d495296a68e2e038b1d212036aeb6424ef366d1006f"} Oct 30 19:29:39.545547 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:39.545456 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podStartSLOduration=3.179268931 podStartE2EDuration="11.545431295s" totalImagesPullingTime="8.366162364s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:39.54280663 +0000 UTC m=+11.705137403" watchObservedRunningTime="2026-10-30 19:29:39.545431295 +0000 UTC m=+11.707762072" Oct 30 19:29:40.360598 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.360572 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:40.360736 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:40.360711 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:40.360820 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.360792 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:40.360869 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:40.360859 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:40.523882 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.523812 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2pmgc" event={"ID":"29bace5a-fb35-45e9-a2ce-41a61af06524","Type":"ContainerStarted","Data":"04bff8136173a0a753844897ab0ca520f1c0a05a758d7dfa4bec26fa43c9672d"} Oct 30 19:29:40.526732 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.526706 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="7f59bbcca1cdb218bd1ef367cab3b09eb35eb96edf77346a3bb38e1dcec7d858" exitCode=0 Oct 30 19:29:40.526858 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.526753 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"7f59bbcca1cdb218bd1ef367cab3b09eb35eb96edf77346a3bb38e1dcec7d858"} Oct 30 19:29:40.544280 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.544235 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2pmgc" podStartSLOduration=4.749936311 podStartE2EDuration="12.544223944s" totalImagesPullingTime="7.794287633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:40.544080412 +0000 UTC m=+12.706411189" watchObservedRunningTime="2026-10-30 19:29:40.544223944 +0000 UTC m=+12.706554720" Oct 30 19:29:40.544655 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:40.544629 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-jcmd8" podStartSLOduration=4.485236593 podStartE2EDuration="12.544622451s" totalImagesPullingTime="8.059385858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:39.59091542 +0000 UTC m=+11.753246198" watchObservedRunningTime="2026-10-30 19:29:40.544622451 +0000 UTC m=+12.706953236" Oct 30 19:29:41.359934 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:41.359898 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:41.360129 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:41.360043 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:41.531711 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:41.531662 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" event={"ID":"e8b76e60-de88-4c65-b570-fb4772dfbb5b","Type":"ContainerStarted","Data":"f76d5fbbc4e465393f6546a4c1d52193d751b5749c0609e96df5efa7b2d456f1"} Oct 30 19:29:41.555449 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:41.555401 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" podStartSLOduration=3.369930914 podStartE2EDuration="13.555389462s" totalImagesPullingTime="10.185458548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:41.555004333 +0000 UTC m=+13.717335109" watchObservedRunningTime="2026-10-30 19:29:41.555389462 +0000 UTC m=+13.717720238" Oct 30 19:29:42.360252 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:42.360224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:42.360376 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:42.360347 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:42.360989 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:42.360965 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:42.361128 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:42.361113 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:42.537346 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:42.537265 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="8bd1f31e71ee20c09e5b5711a172ecc220a8248da68857c12e72efe3d7b7c601" exitCode=0 Oct 30 19:29:42.537897 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:42.537348 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"8bd1f31e71ee20c09e5b5711a172ecc220a8248da68857c12e72efe3d7b7c601"} Oct 30 19:29:43.360441 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:43.360207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:43.360626 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:43.360563 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:43.543773 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:43.543742 2583 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="8bbd1e2709c1f081d0a7f50849013179d388deed72ac245aba44be6a112bcc9e" exitCode=0 Oct 30 19:29:43.544189 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:43.543786 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerDied","Data":"8bbd1e2709c1f081d0a7f50849013179d388deed72ac245aba44be6a112bcc9e"} Oct 30 19:29:44.363762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:44.363731 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:44.363926 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:44.363852 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:44.363926 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:44.363890 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:44.364022 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:44.363951 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:44.549231 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:44.549196 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-l89fd" event={"ID":"651c14dc-17ea-4608-b927-ba1af3ea8b41","Type":"ContainerStarted","Data":"c82032e4dc2efd7c6d5626be74758c41e56346e78aa3190eedefb175615ab893"} Oct 30 19:29:44.581582 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:44.581539 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-l89fd" podStartSLOduration=6.274481989 podStartE2EDuration="16.581526547s" totalImagesPullingTime="10.307044558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:44.581495522 +0000 UTC m=+16.743826300" watchObservedRunningTime="2026-10-30 19:29:44.581526547 +0000 UTC m=+16.743857317" Oct 30 19:29:45.360340 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.360181 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:45.360467 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:45.360423 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:45.554790 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554757 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"5a4b17aa352a6d2f9cd8408714f2fdc2d0160090d272d5197e7f006717ac672f"} Oct 30 19:29:45.555162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554828 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"75bcce6bd61d9ff4db74bbe5d9cd9d2bb695362d038515815491082d6ebf2c60"} Oct 30 19:29:45.555162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554859 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"d3d8ab96d12f29ce65357ea277ef16610ab6d4a524f3662260da10934b2da147"} Oct 30 19:29:45.555162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554890 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"0fabccf8728cc8aa3d3abd9c90a13253aa5a98edf3ee6dffcedd4633169294ae"} Oct 30 19:29:45.555162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554916 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"96846869c4c110b3b524282e3fbc712171359daff9cc8e9839978653b8fd04ff"} Oct 30 19:29:45.555162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.554946 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"241e5980c531b6dff094961b5ab17e4ef810306e866d44a830b8743d2aaa4152"} Oct 30 19:29:45.910279 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:45.910235 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:45.910426 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:45.910389 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:45.910461 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:45.910452 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:30:01.910436728 +0000 UTC m=+34.072767495 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:29:46.011467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:46.011439 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:46.011571 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.011542 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:46.011623 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.011590 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.011578202 +0000 UTC m=+34.173908956 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:29:46.112231 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:46.112202 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:46.112348 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.112328 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:29:46.112348 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.112340 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:29:46.112348 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.112349 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:46.112440 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.112389 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:02.112379474 +0000 UTC m=+34.274710241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:29:46.360904 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:46.360819 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:46.361074 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.360929 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:46.361074 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:46.360977 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:46.361193 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:46.361069 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:47.360732 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:47.360688 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:47.361261 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:47.360834 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:47.562699 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:47.562664 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"d70965fce8c029793c8e7a5d1f5c9dafa195166722bb75f4ce8937443da80d14"} Oct 30 19:29:48.360833 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:48.360797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:48.361200 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:48.360877 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:48.361200 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:48.360928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:48.361200 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:48.361024 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:48.479523 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:48.479488 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:48.682321 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:48.682226 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:49.360422 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:49.360389 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:49.360570 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:49.360486 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:50.361254 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.360991 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:50.361914 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.361028 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:50.361914 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:50.361386 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:50.361914 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:50.361484 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:50.576894 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.576853 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" event={"ID":"b36aff52-af76-477f-990d-9900e4c43954","Type":"ContainerStarted","Data":"7fa58891981d13d5fc917b1f6cd9d75e75e2f97db3a7c0e9ff575ef0a6950818"} Oct 30 19:29:50.577184 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.577160 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:50.577336 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.577232 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:50.593127 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.593104 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:50.593224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.593201 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:50.616248 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:50.616181 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" podStartSLOduration=8.088774584 podStartE2EDuration="22.61616972s" totalImagesPullingTime="14.527395136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:29:50.615640422 +0000 UTC m=+22.777971198" watchObservedRunningTime="2026-10-30 19:29:50.61616972 +0000 UTC m=+22.778500496" Oct 30 19:29:51.263144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.261748 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jllfr"] Oct 30 19:29:51.263144 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.261917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:51.263144 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:51.262132 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:51.264866 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.264831 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrpl8"] Oct 30 19:29:51.265046 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.265026 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:51.265257 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:51.265227 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:51.266084 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.266035 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nnkd"] Oct 30 19:29:51.266235 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.266208 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:51.266538 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:51.266522 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:51.372616 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.372586 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.519113 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.518997 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:51.578746 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.578714 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:29:51.660334 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:51.660300 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.059299 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:52.059266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.204230 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:52.204198 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:52.580255 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:52.580233 2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:29:53.310476 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.310448 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:29:53.360134 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.360113 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:53.360228 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.360191 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:53.360228 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:53.360199 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:53.360228 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.360214 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:53.360316 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:53.360288 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:53.360378 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:53.360362 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:53.464185 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.464160 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:53.595825 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:53.595743 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.411327 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:54.411290 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.553715 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:54.553623 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:54.862553 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:54.862438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.051145 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.051115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.201665 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.201596 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.338086 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.338037 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.360087 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.360052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:55.360087 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.360078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:55.360172 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.360078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:55.360172 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:55.360162 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:55.360248 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:55.360229 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:55.360312 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:55.360298 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:55.577769 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.577695 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:55.715408 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:55.715379 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:56.505446 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:56.505415 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:56.830788 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:56.830703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:56.977317 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:56.977279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.122098 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.122003 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.283826 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.283798 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.360933 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.360901 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:57.360933 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.360926 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:57.361158 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.360941 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:57.361158 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:57.361004 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:57.361158 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:57.361096 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:57.361287 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:57.361203 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:57.492179 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.492112 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:57.593777 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.593754 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jcmd8_ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/dns-node-resolver/0.log" Oct 30 19:29:57.949149 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:57.948910 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.089417 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:58.089383 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.194027 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:58.193998 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gh2g_ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/node-ca/0.log" Oct 30 19:29:58.504091 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:58.504049 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.716559 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:58.716528 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:58.917981 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:58.917916 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:59.115880 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.115848 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:59.314880 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.314813 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:29:59.360678 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.360653 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:29:59.360678 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.360662 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:29:59.360834 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:59.360784 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:29:59.360834 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.360829 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:29:59.360906 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:59.360886 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:29:59.360963 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:29:59.360948 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:29:59.456325 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:29:59.456300 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:00.508937 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:00.508909 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:00.673073 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:00.673025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:00.806345 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:00.806278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:00.943401 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:00.943370 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:01.091104 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.091025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:01.230412 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.230381 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:01.360725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.360693 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:01.360725 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.360716 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:01.360956 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.360791 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:01.360956 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:01.360814 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:01.360956 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:01.360872 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:01.360956 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:01.360942 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:01.382150 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.382101 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:01.921657 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:01.921601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:01.922190 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:01.921748 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:01.922190 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:01.921810 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret podName:56834a9b-e103-4f55-b54f-14e73b4fa1db nodeName:}" failed. No retries permitted until 2026-10-30 19:30:33.921795637 +0000 UTC m=+66.084126391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret") pod "global-pull-secret-syncer-jllfr" (UID: "56834a9b-e103-4f55-b54f-14e73b4fa1db") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:02.022211 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.022177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:02.022388 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.022293 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:02.022388 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.022346 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs podName:33381540-cec9-4797-a8b9-be05dd1f13c0 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:34.022333252 +0000 UTC m=+66.184664019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs") pod "network-metrics-daemon-8nnkd" (UID: "33381540-cec9-4797-a8b9-be05dd1f13c0") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:02.123289 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.123252 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:02.123463 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.123421 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:02.123463 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.123448 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:02.123463 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.123461 2583 projected.go:196] Error preparing data for projected volume kube-api-access-bgkcf for pod openshift-network-diagnostics/network-check-target-mrpl8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:02.123564 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:02.123510 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf podName:ce385369-c46a-4bdd-800a-8ac5ae39a471 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:34.123497054 +0000 UTC m=+66.285827825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-bgkcf" (UniqueName: "kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf") pod "network-check-target-mrpl8" (UID: "ce385369-c46a-4bdd-800a-8ac5ae39a471") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:02.438043 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.438006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:02.577140 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.577103 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:02.712588 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.712513 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:02.905748 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:02.905716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:03.039934 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.039863 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:03.181038 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.181005 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:03.321369 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.321286 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:03.360518 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.360477 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:03.360701 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.360536 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:03.360701 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.360486 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:03.360701 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:03.360604 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:03.360701 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:03.360655 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:03.360901 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:03.360703 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:03.474680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:03.474649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.320781 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:04.320751 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.479478 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:04.479444 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.617499 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:04.617467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.776399 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:04.776342 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:04.953002 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:04.952924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:05.164197 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.164166 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:05.301694 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.301610 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:05.360412 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.360370 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:05.360817 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.360470 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:05.360817 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:05.360480 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:05.360817 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.360499 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:05.360817 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:05.360578 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:05.360817 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:05.360646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:05.719619 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:05.719582 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.325916 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:06.325884 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.461452 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:06.461419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.621175 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:06.621125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:06.911215 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:06.911133 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.108294 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.108258 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.360886 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.360847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:07.361090 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.360862 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:07.361090 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:07.360973 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:07.361090 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.360999 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:07.361260 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:07.361133 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:07.361322 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:07.361291 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:07.709218 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.709144 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:07.910859 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:07.910827 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.109329 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:08.109296 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:08.455918 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:08.455839 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:09.360103 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:09.360052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:09.360103 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:09.360104 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:09.360622 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:09.360170 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:09.360622 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:09.360185 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:09.360622 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:09.360286 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:09.360622 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:09.360375 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:11.360829 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:11.360794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:11.361282 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:11.360833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:11.361282 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:11.360890 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:11.361282 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:11.360943 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:11.361282 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:11.361019 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:11.361282 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:11.361127 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:13.360591 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:13.360558 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:13.361015 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:13.360659 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:13.361015 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:13.360670 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:13.361015 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:13.360743 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:13.361015 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:13.360753 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:13.361015 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:13.360849 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:15.360766 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:15.360724 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:15.361224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:15.360807 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:15.361224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:15.360895 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:15.361224 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:15.360904 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrpl8" podUID="ce385369-c46a-4bdd-800a-8ac5ae39a471" Oct 30 19:30:15.361224 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:15.360960 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jllfr" podUID="56834a9b-e103-4f55-b54f-14e73b4fa1db" Oct 30 19:30:15.361224 ip-10-0-139-103 kubenswrapper[2583]: E1030 19:30:15.361072 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8nnkd" podUID="33381540-cec9-4797-a8b9-be05dd1f13c0" Oct 30 19:30:16.521654 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.521604 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.601890 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.601867 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeReady" Oct 30 19:30:16.602033 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.601992 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:30:16.666313 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.666286 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bct4d"] Oct 30 19:30:16.667995 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.667970 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.676285 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.676262 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\"" Oct 30 19:30:16.681179 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.681153 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:30:16.681325 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.681257 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:30:16.681433 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.681416 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:30:16.682361 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.682339 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bjt85"] Oct 30 19:30:16.684158 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.684139 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bct4d"] Oct 30 19:30:16.684265 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.684221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.699301 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.699285 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:30:16.708159 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.708137 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bjt85"] Oct 30 19:30:16.714766 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.714746 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nrp79"] Oct 30 19:30:16.716559 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.716534 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.716652 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.716564 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.730515 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.730496 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrp79"] Oct 30 19:30:16.732494 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.732475 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:30:16.732805 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.732786 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:30:16.732945 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.732887 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:30:16.779628 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.779597 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\"" Oct 30 19:30:16.779762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.779649 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:30:16.779762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.779725 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:30:16.779762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.779728 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\"" Oct 30 19:30:16.828989 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.828959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.829162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.828995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.829162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6sq\" (UniqueName: \"kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.829162 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829123 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.829333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829183 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.829333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829235 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f2qt\" (UniqueName: \"kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.829333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.829333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829292 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.829333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829322 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpc2r\" (UniqueName: \"kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.829484 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829396 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.829484 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.829436 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.930107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930091 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.930107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6sq\" (UniqueName: \"kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930223 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2qt\" (UniqueName: \"kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930475 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930523 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpc2r\" (UniqueName: \"kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930627 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930636 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.930850 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.931395 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.930972 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.934260 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.934240 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.934260 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.934249 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.934419 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.934288 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.936970 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.936949 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:16.943241 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.943216 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6sq\" (UniqueName: \"kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:16.944227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.944208 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpc2r\" (UniqueName: \"kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:16.944551 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.944533 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2qt\" (UniqueName: \"kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.977516 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.977479 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:16.994874 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:16.994851 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:17.026864 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.026836 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:17.074844 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.074809 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:17.262459 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.262383 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bct4d"] Oct 30 19:30:17.265484 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:17.265456 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0678842b_b564_4b68_ae5b_aaffd1680479.slice/crio-590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983 WatchSource:0}: Error finding container 590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983: Status 404 returned error can't find the container with id 590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983 Oct 30 19:30:17.271015 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.270992 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bjt85"] Oct 30 19:30:17.273351 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:17.273318 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb107b211_0176_4170_8d0f_85d4b36a4f30.slice/crio-b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3 WatchSource:0}: Error finding container b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3: Status 404 returned error can't find the container with id b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3 Oct 30 19:30:17.329201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.329170 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrp79"] Oct 30 19:30:17.332089 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:17.332047 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72fe1029_ecc8_4b99_9387_659ae6e414b3.slice/crio-9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec WatchSource:0}: Error finding container 9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec: Status 404 returned error can't find the container with id 9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec Oct 30 19:30:17.360908 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.360887 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:17.361012 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.360955 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:17.361012 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.360981 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:17.366176 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.366154 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:30:17.366176 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.366154 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:30:17.368116 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.368097 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\"" Oct 30 19:30:17.368224 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.368143 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\"" Oct 30 19:30:17.368395 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.368376 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:30:17.368451 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.368395 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\"" Oct 30 19:30:17.368451 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.368419 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:30:17.632964 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.632931 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrp79" event={"ID":"72fe1029-ecc8-4b99-9387-659ae6e414b3","Type":"ContainerStarted","Data":"bc1d58b24fb647ad9004becac7b065755039ce03dcfab5f9f09fba3e54c17268"} Oct 30 19:30:17.633399 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.632987 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrp79" event={"ID":"72fe1029-ecc8-4b99-9387-659ae6e414b3","Type":"ContainerStarted","Data":"9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec"} Oct 30 19:30:17.634118 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.634087 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjt85" event={"ID":"b107b211-0176-4170-8d0f-85d4b36a4f30","Type":"ContainerStarted","Data":"b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3"} Oct 30 19:30:17.635141 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:17.635114 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bct4d" event={"ID":"0678842b-b564-4b68-ae5b-aaffd1680479","Type":"ContainerStarted","Data":"590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983"} Oct 30 19:30:18.611314 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:18.611229 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:18.640639 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:18.640602 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrp79" event={"ID":"72fe1029-ecc8-4b99-9387-659ae6e414b3","Type":"ContainerStarted","Data":"91c43ce826dbfd483300be09b2e75a85fe0eed39e53447085a5a1cc8cd22106d"} Oct 30 19:30:18.764293 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:18.764210 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.150514 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.150477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.537365 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.537277 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.646403 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.646359 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjt85" event={"ID":"b107b211-0176-4170-8d0f-85d4b36a4f30","Type":"ContainerStarted","Data":"23dde1f6360d4a8d5187cbc547255901e35a2f4481a668674eeab1624a0a868d"} Oct 30 19:30:19.646854 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.646434 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjt85" event={"ID":"b107b211-0176-4170-8d0f-85d4b36a4f30","Type":"ContainerStarted","Data":"e45e18d2836cd828e4c5a4186ac9075fb789ff0d6b902e9be8f887ab9bede091"} Oct 30 19:30:19.647215 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.647189 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:19.675399 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.675307 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-bjt85" podStartSLOduration=1.693462236 podStartE2EDuration="3.675289029s" totalImagesPullingTime="1.981826793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:19.672996976 +0000 UTC m=+51.835327753" watchObservedRunningTime="2026-10-30 19:30:19.675289029 +0000 UTC m=+51.837619868" Oct 30 19:30:19.688005 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.687971 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.843171 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.843132 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:19.996473 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:19.996384 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:20.650618 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.650579 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bct4d" event={"ID":"0678842b-b564-4b68-ae5b-aaffd1680479","Type":"ContainerStarted","Data":"ec94754ee82d6defaf0047d051bdd899dbf18494f3cf051263123fb705054306"} Oct 30 19:30:20.652587 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.652558 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrp79" event={"ID":"72fe1029-ecc8-4b99-9387-659ae6e414b3","Type":"ContainerStarted","Data":"adee9f5512888cc19f45488651309e85319a15fcab38306a18954420bac0acc6"} Oct 30 19:30:20.683613 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.683560 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bct4d" podStartSLOduration=1.748704059 podStartE2EDuration="4.683546521s" totalImagesPullingTime="2.934842462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:20.682973874 +0000 UTC m=+52.845304650" watchObservedRunningTime="2026-10-30 19:30:20.683546521 +0000 UTC m=+52.845877298" Oct 30 19:30:20.709470 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.709405 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nrp79" podStartSLOduration=2.454169952 podStartE2EDuration="4.70939045s" totalImagesPullingTime="2.255220498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:20.708498788 +0000 UTC m=+52.870829565" watchObservedRunningTime="2026-10-30 19:30:20.70939045 +0000 UTC m=+52.871721241" Oct 30 19:30:20.739321 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.739292 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:20.873036 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:20.873000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.135358 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:21.135327 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.273286 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:21.273254 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.414737 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:21.414667 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:21.557700 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:21.557667 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.139271 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:22.139238 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.273629 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:22.273598 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.529663 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:22.529579 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:22.662885 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:22.662856 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:23.593449 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:23.593421 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:23.864216 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:23.864189 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.049113 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.049078 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.140138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.140041 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf"] Oct 30 19:30:24.142125 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.142106 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.152454 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.152437 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:30:24.156152 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.156131 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:30:24.156279 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.156255 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qv2b2\"" Oct 30 19:30:24.158730 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.158581 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:30:24.158730 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.158604 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:30:24.158851 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.158768 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:30:24.160420 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.160398 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf"] Oct 30 19:30:24.173109 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.173090 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g"] Oct 30 19:30:24.174902 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.174888 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.195448 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.195430 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g"] Oct 30 19:30:24.196108 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.196088 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tx4kh"] Oct 30 19:30:24.197863 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.197847 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.212214 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.212194 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:30:24.212214 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.212201 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:30:24.212347 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.212202 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\"" Oct 30 19:30:24.212347 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.212247 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pr8mc\"" Oct 30 19:30:24.214680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.214662 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:30:24.214890 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.214875 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:30:24.239627 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.239603 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.255136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.255118 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:30:24.255225 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.255173 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:30:24.284818 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.284789 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.284909 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.284822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.284909 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.284853 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.285003 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.284940 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.285003 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.284988 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285077 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.285121 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.285121 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285199 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285136 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.285244 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285193 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285244 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285219 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285303 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285254 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6vj\" (UniqueName: \"kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285341 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285302 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285341 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285328 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285434 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285351 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.285434 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285399 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285528 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.285528 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285460 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65djh\" (UniqueName: \"kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.285528 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.285485 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc22l\" (UniqueName: \"kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.386269 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386239 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386440 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386278 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6vj\" (UniqueName: \"kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386440 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386316 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386440 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386440 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.386440 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386427 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386476 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386503 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65djh\" (UniqueName: \"kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc22l\" (UniqueName: \"kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386584 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386616 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.386696 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386657 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386708 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386733 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386841 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386881 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.386923 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387024 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387009 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387410 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387155 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387410 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387159 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.387410 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387231 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387410 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387268 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387742 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387715 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.387885 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387866 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.387953 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387917 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.387953 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.387935 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.389113 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.389084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.389262 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.389245 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.389559 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.389535 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.390251 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.390199 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.390322 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.390269 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.390399 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.390382 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.402376 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.402330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.404227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.404195 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6vj\" (UniqueName: \"kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.404317 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.404237 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65djh\" (UniqueName: \"kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.411229 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.411207 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc22l\" (UniqueName: \"kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.455049 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.455001 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:24.486841 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.486809 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:24.508424 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.508391 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:24.523836 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:24.523808 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7a804_0740_4d51_884b_ea661daf80be.slice/crio-339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a WatchSource:0}: Error finding container 339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a: Status 404 returned error can't find the container with id 339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a Oct 30 19:30:24.542949 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.542866 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.663193 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.663116 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tx4kh" event={"ID":"a1c7a804-0740-4d51-884b-ea661daf80be","Type":"ContainerStarted","Data":"339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a"} Oct 30 19:30:24.729472 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.729446 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf"] Oct 30 19:30:24.731640 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:24.731618 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d4ab99_d6b0_4b6c_99ed_ee24feac5df0.slice/crio-c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2 WatchSource:0}: Error finding container c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2: Status 404 returned error can't find the container with id c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2 Oct 30 19:30:24.733728 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.733705 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:24.776424 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.776399 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g"] Oct 30 19:30:24.778526 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:24.778503 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87433f0_7272_49cb_a84e_20565709bdae.slice/crio-bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1 WatchSource:0}: Error finding container bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1: Status 404 returned error can't find the container with id bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1 Oct 30 19:30:24.878085 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:24.878045 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.148049 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.148014 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.310581 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.310546 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.318626 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.317263 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:25.319810 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.319785 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.325782 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.325761 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:30:25.326262 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.326093 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:30:25.330069 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.330035 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.330225 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.331104 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.331397 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.331780 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.332232 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.332495 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:30:25.334749 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.332640 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wsp2p\"" Oct 30 19:30:25.339179 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.339050 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:25.483765 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.483682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.497566 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497513 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497571 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497612 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497680 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497670 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497806 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497806 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497806 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497731 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497806 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497767 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497806 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497791 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497941 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497810 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497941 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.497941 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.497856 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599051 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599201 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599271 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599293 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599353 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.599467 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599441 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.600175 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.599859 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.600668 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.600646 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.602317 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.601967 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.603107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.603084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.604152 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.604129 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.609495 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.609472 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.610384 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.609624 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.610384 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.609990 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.610384 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.610350 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.610959 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.610934 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.611049 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.610941 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.611257 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.611235 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.616278 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.616233 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.621107 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.621044 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:25.651846 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.651812 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:25.669095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.668526 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" event={"ID":"d87433f0-7272-49cb-a84e-20565709bdae","Type":"ContainerStarted","Data":"bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1"} Oct 30 19:30:25.671267 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.671127 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" event={"ID":"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0","Type":"ContainerStarted","Data":"499de256132fa4425f51a4503fc1154082270ed7204556c327f4a69a2c87134f"} Oct 30 19:30:25.671267 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.671198 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" event={"ID":"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0","Type":"ContainerStarted","Data":"43f3b6b3f2b72e40d73e04a0a3bece720e3342a319fc9e85350ad757d195fd3d"} Oct 30 19:30:25.671267 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.671229 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" event={"ID":"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0","Type":"ContainerStarted","Data":"c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2"} Oct 30 19:30:25.673335 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.673297 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tx4kh" event={"ID":"a1c7a804-0740-4d51-884b-ea661daf80be","Type":"ContainerStarted","Data":"f83f180862c3048f480a2ef67ed067b08572e380d22c880b1752516dbdae3304"} Oct 30 19:30:25.952237 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:25.952205 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:25.953263 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:25.953235 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cf1b54_38ba_446d_8083_b66ba618a7db.slice/crio-bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4 WatchSource:0}: Error finding container bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4: Status 404 returned error can't find the container with id bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4 Oct 30 19:30:26.504228 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.504149 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:30:26.506374 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.506338 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.520250 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.520223 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:30:26.522733 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.522708 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:30:26.522831 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.522748 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6jgjw\"" Oct 30 19:30:26.522831 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.522816 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:30:26.522957 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.522867 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:30:26.522957 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.522891 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:30:26.524088 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.524068 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:30:26.533954 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.533928 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:30:26.534049 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.534011 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 19:30:26.534166 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.534138 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:30:26.610456 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610419 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610610 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610610 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610715 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610651 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610762 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610751 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610855 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610775 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.610905 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.610884 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.679729 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.679536 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" event={"ID":"d87433f0-7272-49cb-a84e-20565709bdae","Type":"ContainerStarted","Data":"81a9f1f560ece4e322885a531e960d716b3e3a115631be36d2fde4dfe2b2d93e"} Oct 30 19:30:26.679729 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.679642 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" event={"ID":"d87433f0-7272-49cb-a84e-20565709bdae","Type":"ContainerStarted","Data":"dec3f85b8f980caa9d1733119e06e9872fd2469ab6b6e53d1992357389f06cb6"} Oct 30 19:30:26.679729 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.679674 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" event={"ID":"d87433f0-7272-49cb-a84e-20565709bdae","Type":"ContainerStarted","Data":"99c948f2f8888d323141932c7893c898daff561a6435d273dabfec129cc76827"} Oct 30 19:30:26.682363 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.682330 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" event={"ID":"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0","Type":"ContainerStarted","Data":"003129dfa58700363d593ed82d4a52e81e657dd7a75878d32f4555bd9d470712"} Oct 30 19:30:26.685105 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.685079 2583 generic.go:356] "Generic (PLEG): container finished" podID="a1c7a804-0740-4d51-884b-ea661daf80be" containerID="f83f180862c3048f480a2ef67ed067b08572e380d22c880b1752516dbdae3304" exitCode=0 Oct 30 19:30:26.685214 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.685155 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tx4kh" event={"ID":"a1c7a804-0740-4d51-884b-ea661daf80be","Type":"ContainerDied","Data":"f83f180862c3048f480a2ef67ed067b08572e380d22c880b1752516dbdae3304"} Oct 30 19:30:26.686596 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.686570 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4"} Oct 30 19:30:26.705451 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.705237 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" podStartSLOduration=1.343636036 podStartE2EDuration="2.70521877s" totalImagesPullingTime="1.361582734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:26.703716232 +0000 UTC m=+58.866047009" watchObservedRunningTime="2026-10-30 19:30:26.70521877 +0000 UTC m=+58.867549548" Oct 30 19:30:26.711230 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711207 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711324 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711259 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711324 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711324 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711318 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711505 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711346 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711505 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711373 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711505 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711419 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.711505 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.711446 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.712075 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.712041 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.712461 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.712433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.712576 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.712552 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.712748 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.712729 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.712947 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.712928 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.713939 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.713914 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.714041 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.714024 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.724824 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.724804 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.726727 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.726703 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:26.731881 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.731837 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" podStartSLOduration=1.6635299890000002 podStartE2EDuration="2.7318214s" totalImagesPullingTime="1.068291411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:26.730604615 +0000 UTC m=+58.892935391" watchObservedRunningTime="2026-10-30 19:30:26.7318214 +0000 UTC m=+58.894152177" Oct 30 19:30:26.819117 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.819015 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:26.871447 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:26.871420 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.101450 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.101419 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:30:27.105871 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:27.105850 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6539c7b_9167_4d62_a086_6c02a73827d2.slice/crio-78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31 WatchSource:0}: Error finding container 78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31: Status 404 returned error can't find the container with id 78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31 Oct 30 19:30:27.226114 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.226044 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.362044 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.362004 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.519106 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.519052 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:27.693517 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.693436 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tx4kh" event={"ID":"a1c7a804-0740-4d51-884b-ea661daf80be","Type":"ContainerStarted","Data":"0ad79950eed6ac4ddf3ee71f66be44bc76734b8d94979ac2108b5c118d17c397"} Oct 30 19:30:27.693517 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.693493 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tx4kh" event={"ID":"a1c7a804-0740-4d51-884b-ea661daf80be","Type":"ContainerStarted","Data":"962ad524890910468d24671cefb76cd53be0e5a84a8bc844dcf7ca7cf89be83e"} Oct 30 19:30:27.695792 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.695757 2583 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d" exitCode=0 Oct 30 19:30:27.695931 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.695850 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d"} Oct 30 19:30:27.698043 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.698021 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6864546c55-lv6zs" event={"ID":"c6539c7b-9167-4d62-a086-6c02a73827d2","Type":"ContainerStarted","Data":"78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31"} Oct 30 19:30:27.724879 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:27.724836 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tx4kh" podStartSLOduration=2.6013365090000002 podStartE2EDuration="3.724821898s" totalImagesPullingTime="1.027271633s" totalInitContainerRuntime="96.213756ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:27.724489734 +0000 UTC m=+59.886820512" watchObservedRunningTime="2026-10-30 19:30:27.724821898 +0000 UTC m=+59.887152668" Oct 30 19:30:28.595942 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.595901 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:28.615165 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.615127 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bd8695446-nbt67"] Oct 30 19:30:28.617587 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.617565 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.622596 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.622571 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:30:28.622720 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.622652 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:30:28.630631 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.630415 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd8695446-nbt67"] Oct 30 19:30:28.636411 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.636389 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:30:28.636552 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.636536 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:30:28.636687 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.636664 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8apo69g0vql64\"" Oct 30 19:30:28.636808 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.636794 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jqkdh\"" Oct 30 19:30:28.730330 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtnk\" (UniqueName: \"kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730370 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730528 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730573 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730617 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.730734 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.730704 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.739372 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.739339 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831638 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831721 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831767 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833095 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.831920 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtnk\" (UniqueName: \"kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833571 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.833182 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833721 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.833688 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.833786 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.833730 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.837084 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.835564 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.837084 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.835731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.837084 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.835757 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.846823 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.846759 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtnk\" (UniqueName: \"kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:28.936741 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:28.936703 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:29.098545 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.098464 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.223077 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.223021 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bd8695446-nbt67"] Oct 30 19:30:29.226711 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:29.226673 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38afb684_e999_4c10_9281_62f633431cc6.slice/crio-77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91 WatchSource:0}: Error finding container 77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91: Status 404 returned error can't find the container with id 77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91 Oct 30 19:30:29.252073 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.251944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:29.707986 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.707888 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa"} Oct 30 19:30:29.707986 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.707978 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac"} Oct 30 19:30:29.708212 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.708017 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909"} Oct 30 19:30:29.708212 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.708074 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5"} Oct 30 19:30:29.708212 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.708114 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7"} Oct 30 19:30:29.709302 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:29.709272 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" event={"ID":"38afb684-e999-4c10-9281-62f633431cc6","Type":"ContainerStarted","Data":"77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91"} Oct 30 19:30:30.532030 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.531995 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:30:30.534795 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.534767 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.540971 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.540947 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:30:30.541116 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.540966 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:30:30.541116 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.540965 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:30:30.541116 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.540998 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:30:30.541244 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.541152 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:30:30.541244 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.541183 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:30:30.541359 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.541282 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6b35nveaduvtq\"" Oct 30 19:30:30.541429 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.541408 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:30:30.541473 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.541460 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:30:30.548960 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.548941 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:30:30.549151 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.549033 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:30:30.549151 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.549078 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:30:30.549262 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.549171 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:30:30.549333 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.549308 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5kn8\"" Oct 30 19:30:30.555511 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.554590 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:30:30.557502 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.557479 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:30:30.559431 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.559403 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:30:30.565966 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.565904 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:30.648282 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648245 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648419 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648285 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648419 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648529 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648462 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648585 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648550 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648635 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648595 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648635 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648620 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648700 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648669 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648741 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648699 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648741 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648719 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648818 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648798 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648882 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648864 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648929 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648982 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648942 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.648982 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.648970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.649097 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.649026 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.649097 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.649048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.649204 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.649173 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.649339 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.649213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.649339 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.649268 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.658382 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.658361 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:30.719841 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.719808 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerStarted","Data":"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7"} Oct 30 19:30:30.722800 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.722748 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6864546c55-lv6zs" event={"ID":"c6539c7b-9167-4d62-a086-6c02a73827d2","Type":"ContainerStarted","Data":"047baa3ddc2d1809f035e106139e5a0b8132291dbfe5fd321e445245535dc3e0"} Oct 30 19:30:30.749968 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.749927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.749978 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750009 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750070 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750138 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750131 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750160 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750192 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750249 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750274 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750344 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750386 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750422 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750447 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750554 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.750636 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.750601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.752133 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.751130 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.752133 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.751682 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.752133 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.751829 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.753021 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.752996 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.754019 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.753989 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.754019 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.754010 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.754606 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.754559 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.755436 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.755409 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.756928 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.755826 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.756928 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.756822 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.757947 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.757926 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.758029 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.757949 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.4210493189999998 podStartE2EDuration="5.757937149s" totalImagesPullingTime="3.324907634s" totalInitContainerRuntime="11.980196ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:30.756021188 +0000 UTC m=+62.918351963" watchObservedRunningTime="2026-10-30 19:30:30.757937149 +0000 UTC m=+62.920267924" Oct 30 19:30:30.758747 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.758724 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.758747 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.758736 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.759561 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.759538 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.759657 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.759581 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:30.759978 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.759951 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.760068 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.760039 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.760150 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.760127 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.761716 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.761693 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.769240 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.769176 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:30.823533 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.823434 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6864546c55-lv6zs" podStartSLOduration=2.073399504 podStartE2EDuration="4.823422151s" totalImagesPullingTime="2.750022647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:30.821733057 +0000 UTC m=+62.984063860" watchObservedRunningTime="2026-10-30 19:30:30.823422151 +0000 UTC m=+62.985752926" Oct 30 19:30:30.854309 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:30.854268 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:31.149898 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.149867 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:30:31.154164 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:31.154120 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6376030_0944_400b_942f_286dc1cce30b.slice/crio-d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3 WatchSource:0}: Error finding container d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3: Status 404 returned error can't find the container with id d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3 Oct 30 19:30:31.479072 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.479031 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.626610 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.626576 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:31.727772 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.727734 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" event={"ID":"38afb684-e999-4c10-9281-62f633431cc6","Type":"ContainerStarted","Data":"e91cda68bca3bac20fe06da4495966a7f7b1e4b1c44993f8a6c43a40a9825f63"} Oct 30 19:30:31.729388 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.729306 2583 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" exitCode=0 Oct 30 19:30:31.729388 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.729358 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} Oct 30 19:30:31.729534 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.729430 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3"} Oct 30 19:30:31.752020 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:31.751978 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podStartSLOduration=1.9485824250000001 podStartE2EDuration="3.751966359s" totalImagesPullingTime="1.803383934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:31.750931135 +0000 UTC m=+63.913261911" watchObservedRunningTime="2026-10-30 19:30:31.751966359 +0000 UTC m=+63.914297125" Oct 30 19:30:32.936588 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:32.936527 2583 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:30:33.980534 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:33.980490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:33.985136 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:33.985102 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:30:33.993958 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:33.993920 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:34.081037 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.081000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:34.085188 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.085164 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:30:34.094256 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.094225 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:34.178520 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.178491 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\"" Oct 30 19:30:34.181889 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.181865 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:34.184802 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.184782 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:34.185916 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.185893 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:30:34.188677 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.188661 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\"" Oct 30 19:30:34.194945 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.194917 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:34.196708 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.196688 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:30:34.206876 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.206846 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:34.478310 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.478284 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jllfr"] Oct 30 19:30:34.482745 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:34.481759 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56834a9b_e103_4f55_b54f_14e73b4fa1db.slice/crio-8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1 WatchSource:0}: Error finding container 8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1: Status 404 returned error can't find the container with id 8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1 Oct 30 19:30:34.482745 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.482677 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8nnkd"] Oct 30 19:30:34.483202 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.483119 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\"" Oct 30 19:30:34.485665 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:34.485643 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33381540_cec9_4797_a8b9_be05dd1f13c0.slice/crio-1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1 WatchSource:0}: Error finding container 1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1: Status 404 returned error can't find the container with id 1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1 Oct 30 19:30:34.489447 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.489429 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:34.743227 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.743187 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nnkd" event={"ID":"33381540-cec9-4797-a8b9-be05dd1f13c0","Type":"ContainerStarted","Data":"1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1"} Oct 30 19:30:34.744689 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.744625 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jllfr" event={"ID":"56834a9b-e103-4f55-b54f-14e73b4fa1db","Type":"ContainerStarted","Data":"8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1"} Oct 30 19:30:34.767406 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:34.767385 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrpl8"] Oct 30 19:30:34.769590 ip-10-0-139-103 kubenswrapper[2583]: W1030 19:30:34.769563 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce385369_c46a_4bdd_800a_8ac5ae39a471.slice/crio-a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0 WatchSource:0}: Error finding container a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0: Status 404 returned error can't find the container with id a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0 Oct 30 19:30:35.379611 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:35.379461 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:35.580836 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:35.580784 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:35.754150 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:35.754113 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} Oct 30 19:30:35.754282 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:35.754237 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} Oct 30 19:30:35.756046 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:35.756018 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrpl8" event={"ID":"ce385369-c46a-4bdd-800a-8ac5ae39a471","Type":"ContainerStarted","Data":"a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0"} Oct 30 19:30:36.384407 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.384368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.528484 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.528446 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.674587 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.674493 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:36.765307 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.764428 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nnkd" event={"ID":"33381540-cec9-4797-a8b9-be05dd1f13c0","Type":"ContainerStarted","Data":"aa76782b95e510d3ba7c2b4a8a1da26257d1f78b927d5998928e7dd44116b397"} Oct 30 19:30:36.765307 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.764498 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8nnkd" event={"ID":"33381540-cec9-4797-a8b9-be05dd1f13c0","Type":"ContainerStarted","Data":"37a7bbdd5def5177a33923f72ba662c1915123e3b8ce16b3bd4dfa1c7b7eda11"} Oct 30 19:30:36.789176 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.789118 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8nnkd" podStartSLOduration=67.597048691 podStartE2EDuration="1m8.789097183s" totalImagesPullingTime="1.192048492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:36.788758049 +0000 UTC m=+68.951088827" watchObservedRunningTime="2026-10-30 19:30:36.789097183 +0000 UTC m=+68.951427959" Oct 30 19:30:36.819777 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.819739 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:36.984713 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:36.984621 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.121111 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.121006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.272325 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.272240 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.606597 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.606565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.743614 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.743536 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:37.776929 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.776903 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} Oct 30 19:30:37.886196 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:37.886127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:38.768996 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.768894 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:38.790699 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.790665 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} Oct 30 19:30:38.790866 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.790772 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} Oct 30 19:30:38.790866 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.790820 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerStarted","Data":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} Oct 30 19:30:38.792411 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.792385 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrpl8" event={"ID":"ce385369-c46a-4bdd-800a-8ac5ae39a471","Type":"ContainerStarted","Data":"aae9de85d85bfc3c2944381b62d30cacd7c490d792afdf33ffdb75f3f0fd3360"} Oct 30 19:30:38.792540 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.792520 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:38.833823 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.833775 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.046892891 podStartE2EDuration="8.833759473s" totalImagesPullingTime="5.769290921s" totalInitContainerRuntime="17.575661ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:38.830791378 +0000 UTC m=+70.993122147" watchObservedRunningTime="2026-10-30 19:30:38.833759473 +0000 UTC m=+70.996090248" Oct 30 19:30:38.852379 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.852323 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mrpl8" podStartSLOduration=67.179022173 podStartE2EDuration="1m10.852306108s" totalImagesPullingTime="3.673283935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:38.850628179 +0000 UTC m=+71.012958957" watchObservedRunningTime="2026-10-30 19:30:38.852306108 +0000 UTC m=+71.014636884" Oct 30 19:30:38.905930 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:38.905893 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:39.798458 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:39.798414 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jllfr" event={"ID":"56834a9b-e103-4f55-b54f-14e73b4fa1db","Type":"ContainerStarted","Data":"950da6a37e8fd36413614e562ec9d880d189ed25012d8cb5a1125ecdf0869385"} Oct 30 19:30:39.828739 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:39.826887 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jllfr" podStartSLOduration=66.689022018 podStartE2EDuration="1m11.826869745s" totalImagesPullingTime="5.137847727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:39.825683407 +0000 UTC m=+71.988014208" watchObservedRunningTime="2026-10-30 19:30:39.826869745 +0000 UTC m=+71.989200522" Oct 30 19:30:39.876578 ip-10-0-139-103 kubenswrapper[2583]: I1030 19:30:39.876512 2583 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:30:39.876683 ip-10-0-139-103 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:30:39.892467 ip-10-0-139-103 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:30:39.892628 ip-10-0-139-103 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:30:39.892675 ip-10-0-139-103 systemd[1]: kubelet.service: Consumed 4.393s CPU time. Oct 30 19:30:39.901755 ip-10-0-139-103 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:30:39.960301 ip-10-0-139-103 kubenswrapper[5733]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:39.960301 ip-10-0-139-103 kubenswrapper[5733]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:30:39.960301 ip-10-0-139-103 kubenswrapper[5733]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:39.960301 ip-10-0-139-103 kubenswrapper[5733]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:39.962366 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962291 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.962366 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962365 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.962366 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962369 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962372 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962375 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962378 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962381 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962384 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962386 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962389 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962393 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962395 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962398 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962401 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962403 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962410 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962414 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962417 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962419 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962422 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962424 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962427 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.962458 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962430 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962432 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962435 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962437 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962440 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962443 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962446 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962448 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962452 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962454 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962457 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962460 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962463 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962466 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962469 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962472 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962474 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962477 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962480 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962482 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.962927 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962485 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962488 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962490 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962493 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962495 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962498 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962501 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962504 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962506 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962509 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962511 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962514 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962516 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962520 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962523 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962526 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962528 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962531 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962534 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962537 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.963419 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962540 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962543 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962545 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962548 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962551 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962553 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962556 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962558 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962561 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962563 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962566 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962568 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962571 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962573 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962576 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962579 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962581 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962584 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962587 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.963897 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962590 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962594 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962599 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962601 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962604 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962607 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962611 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962613 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962616 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962619 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962622 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962624 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962629 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962633 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962636 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962639 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962642 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962645 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.964378 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962647 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962650 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962652 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962655 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962678 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962683 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.962686 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963428 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963435 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963440 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963445 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963449 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963453 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963455 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963458 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963462 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963464 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963467 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963470 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.964801 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963474 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963477 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963479 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963482 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963485 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963487 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963491 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963493 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963496 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963499 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963502 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963504 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963507 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963510 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963512 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963515 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963518 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963520 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963523 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963525 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.965270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963528 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963530 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963533 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963535 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963538 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963541 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963543 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963546 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963548 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963551 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963558 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963560 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963563 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963566 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963569 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963571 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963574 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963577 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963579 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963582 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.965823 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963584 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963587 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963590 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963593 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963595 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963598 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963600 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963603 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963606 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963608 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963611 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963614 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963616 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963620 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963623 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963626 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963629 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963632 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963635 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.966309 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963638 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963640 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963643 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963647 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963650 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963653 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963656 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963659 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963662 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963664 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963667 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963669 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963672 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963675 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963677 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963680 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963683 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963685 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963688 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963691 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.966765 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963693 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963696 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963698 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963701 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963703 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963707 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963710 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963712 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963715 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963718 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963720 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963723 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963725 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963728 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.963730 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964230 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964236 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964239 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964242 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964245 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.967240 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964248 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964251 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964253 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964256 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964259 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964261 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964264 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964266 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964269 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964272 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964274 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964277 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964280 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964283 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964285 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964287 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964290 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964293 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964295 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.967701 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964298 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964300 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964303 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964305 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964308 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964311 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964313 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964316 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964321 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964324 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964327 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964329 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964332 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964334 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964337 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964340 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964342 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964345 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964347 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964350 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.968201 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964352 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964355 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964358 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964360 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964363 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964366 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964368 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964371 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964374 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964376 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964379 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964382 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964385 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964388 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964390 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964393 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964396 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964398 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964402 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.968671 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964405 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964409 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964412 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964416 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964420 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964424 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964427 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964430 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964433 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964435 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964438 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964441 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964443 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964446 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964449 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964451 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964454 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964456 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964459 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.969129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964461 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964464 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964467 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964471 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964474 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964477 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964480 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964483 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964487 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964490 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964492 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964495 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964497 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964500 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964503 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964506 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964509 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964512 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964515 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964518 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.969585 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964520 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964523 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964525 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.964528 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964620 5733 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964627 5733 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964634 5733 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964638 5733 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964643 5733 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964648 5733 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964656 5733 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964661 5733 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964664 5733 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964667 5733 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964671 5733 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964675 5733 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964678 5733 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964681 5733 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964684 5733 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964688 5733 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964690 5733 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964694 5733 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964698 5733 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964701 5733 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:30:39.970067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964704 5733 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964707 5733 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964712 5733 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964716 5733 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964719 5733 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964722 5733 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964726 5733 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964729 5733 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964732 5733 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964735 5733 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964738 5733 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964743 5733 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964747 5733 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964750 5733 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964754 5733 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964757 5733 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964760 5733 flags.go:64] FLAG: --enable-server="true" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964763 5733 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964771 5733 flags.go:64] FLAG: --event-burst="100" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964774 5733 flags.go:64] FLAG: --event-qps="50" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964777 5733 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964780 5733 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964783 5733 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964787 5733 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:30:39.970629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964791 5733 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964794 5733 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964797 5733 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964801 5733 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964804 5733 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964807 5733 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964810 5733 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964813 5733 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964816 5733 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964819 5733 flags.go:64] FLAG: --feature-gates="" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964824 5733 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964827 5733 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964830 5733 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964834 5733 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964837 5733 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964845 5733 flags.go:64] FLAG: --help="false" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964848 5733 flags.go:64] FLAG: --hostname-override="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964852 5733 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964855 5733 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964858 5733 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964861 5733 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964865 5733 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964867 5733 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:30:39.971242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964870 5733 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964873 5733 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964876 5733 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964879 5733 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964882 5733 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964885 5733 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964888 5733 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964891 5733 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964895 5733 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964898 5733 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964901 5733 flags.go:64] FLAG: --lock-file="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964904 5733 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964907 5733 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964909 5733 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964915 5733 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964918 5733 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964921 5733 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964924 5733 flags.go:64] FLAG: --logging-format="text" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964927 5733 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964930 5733 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964933 5733 flags.go:64] FLAG: --manifest-url="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964936 5733 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964940 5733 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964943 5733 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964947 5733 flags.go:64] FLAG: --max-pods="110" Oct 30 19:30:39.971787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964956 5733 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964959 5733 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964962 5733 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964965 5733 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964968 5733 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964971 5733 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964974 5733 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964981 5733 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964984 5733 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964987 5733 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964990 5733 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964993 5733 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964996 5733 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.964999 5733 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965001 5733 flags.go:64] FLAG: --port="10250" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965006 5733 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965009 5733 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-012b7be2a08857dfd" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965013 5733 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965016 5733 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965019 5733 flags.go:64] FLAG: --register-node="true" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965022 5733 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965026 5733 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965029 5733 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965031 5733 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965034 5733 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965038 5733 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:30:39.972434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965041 5733 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965044 5733 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965048 5733 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965051 5733 flags.go:64] FLAG: --runonce="false" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965068 5733 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965072 5733 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965075 5733 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965086 5733 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965089 5733 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965092 5733 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965095 5733 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965098 5733 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965101 5733 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965104 5733 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965107 5733 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965110 5733 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965113 5733 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965116 5733 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965119 5733 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965125 5733 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965128 5733 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965133 5733 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965139 5733 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965142 5733 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965145 5733 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:30:39.973156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965148 5733 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965150 5733 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965153 5733 flags.go:64] FLAG: --v="2" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965158 5733 flags.go:64] FLAG: --version="false" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965162 5733 flags.go:64] FLAG: --vmodule="" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965166 5733 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965169 5733 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965304 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965309 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965312 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965315 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965318 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965321 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965323 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965326 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965329 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965332 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965334 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965337 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965339 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.973748 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965350 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965353 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965356 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965359 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965364 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965367 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965369 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965372 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965376 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965378 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965381 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965383 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965386 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965389 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965392 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965395 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965397 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965400 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965403 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965405 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.974237 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965408 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965410 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965413 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965415 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965418 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965421 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965425 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965428 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965432 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965435 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965438 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965440 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965443 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965445 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965448 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965451 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965454 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965457 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965460 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965463 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.974717 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965467 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965470 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965472 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965475 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965477 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965480 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965483 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965486 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965488 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965491 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965493 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965496 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965498 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965501 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965503 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965506 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965509 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965511 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965514 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.975203 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965516 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965519 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965522 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965525 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965527 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965530 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965532 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965535 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965538 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965542 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965544 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965547 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965550 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965554 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965557 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965559 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965561 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965564 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965567 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.975894 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965569 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965572 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965575 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965577 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965580 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965584 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965588 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965590 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965593 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965596 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965598 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965601 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965603 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965606 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965609 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.965634 5733 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:39.976613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965670 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965674 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965677 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965680 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965683 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965686 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965688 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965692 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965695 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965698 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965702 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965705 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965707 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965710 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965713 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965716 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965718 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965721 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965725 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965729 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.977223 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965732 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965735 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965737 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965740 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965743 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965745 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965748 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965750 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965753 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965756 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965758 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965761 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965764 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965767 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965769 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965772 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965774 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965777 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965780 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965784 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.977903 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965786 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965789 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965793 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965796 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965799 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965802 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965804 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965807 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965809 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965812 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965815 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965818 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965820 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965823 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965825 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965828 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965831 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965833 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965836 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.978606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965839 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965842 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965845 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965847 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965850 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965853 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965855 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965864 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965867 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965870 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965872 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965875 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965879 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965881 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965884 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965889 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965893 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965897 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965900 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.979196 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965903 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965906 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965908 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965911 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965914 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965916 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965919 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965922 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965925 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965927 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965930 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965932 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965935 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965938 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965940 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965943 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965945 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965948 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965950 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.979655 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965953 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965955 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965963 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965966 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965969 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965975 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965980 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965983 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.965985 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.969798 5733 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.969814 5733 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.969833 5733 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.969837 5733 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969910 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969915 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969919 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969922 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969925 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969928 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969930 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.980127 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969933 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969936 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969939 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969943 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969947 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969950 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969953 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969956 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969958 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969961 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969964 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969966 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969969 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969972 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969975 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969978 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969980 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969983 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969986 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.980647 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969988 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969991 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969994 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.969998 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970001 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970003 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970006 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970009 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970011 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970014 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970017 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970019 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970022 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970025 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970027 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970030 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970033 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970036 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970040 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970042 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.981270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970045 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970048 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970050 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970068 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970072 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970075 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970078 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970081 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970084 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970086 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970089 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970092 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970094 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970097 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970100 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970103 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970106 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970109 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970112 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970114 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.981813 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970117 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970119 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970122 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970125 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970127 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970130 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970133 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970135 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970138 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970140 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970143 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970146 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970148 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970151 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970153 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970156 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970158 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970161 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970163 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.982311 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970166 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970169 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970172 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970175 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970177 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970180 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970182 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970185 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970188 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970190 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970193 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970196 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970198 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970201 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970203 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970206 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970209 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970211 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970214 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970217 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.982798 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970219 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.970247 5733 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970396 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970401 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970404 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970407 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970410 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970413 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970416 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970419 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970421 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970424 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970426 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970430 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970432 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970435 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.983350 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970438 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970440 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970443 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970446 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970448 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970452 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970456 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970463 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970466 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970468 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970471 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970473 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970476 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970478 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970481 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970483 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970486 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970489 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970491 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.983741 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970494 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970496 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970500 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970503 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970505 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970508 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970510 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970513 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970515 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970519 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970523 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970526 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970529 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970532 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970534 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970537 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970540 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970542 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970545 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970547 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.984263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970550 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970553 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970556 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970558 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970561 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970564 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970566 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970569 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970571 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970574 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970577 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970579 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970582 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970584 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970587 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970589 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970592 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970594 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970597 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.984766 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970600 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970602 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970605 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970608 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970611 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970613 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970616 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970619 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970621 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970624 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970626 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970629 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970632 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970635 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970637 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970640 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970643 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970646 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970648 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.985231 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970651 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970653 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970656 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970658 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970661 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970663 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970666 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970668 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970671 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970674 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970676 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970679 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970681 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970684 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970686 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.970709 5733 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:39.985681 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970738 5733 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970742 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970745 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970748 5733 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970751 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970754 5733 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970756 5733 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970759 5733 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970762 5733 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970765 5733 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970767 5733 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970770 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970772 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970778 5733 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970781 5733 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970783 5733 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970786 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970788 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970791 5733 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970793 5733 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:39.986113 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970796 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970798 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970802 5733 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970807 5733 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970810 5733 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970813 5733 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970816 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970818 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970821 5733 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970824 5733 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970826 5733 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970829 5733 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970834 5733 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970837 5733 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970839 5733 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970842 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970844 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970847 5733 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970850 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:39.986595 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970852 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970855 5733 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970858 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970860 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970863 5733 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970865 5733 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970869 5733 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970873 5733 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970876 5733 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970878 5733 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970881 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970883 5733 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970886 5733 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970889 5733 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970891 5733 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970893 5733 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970897 5733 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970899 5733 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970902 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970904 5733 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:39.987051 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970907 5733 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970909 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970913 5733 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970916 5733 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970918 5733 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970922 5733 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970925 5733 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970928 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970930 5733 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970933 5733 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970935 5733 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970938 5733 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970941 5733 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970943 5733 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970946 5733 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970948 5733 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970951 5733 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970953 5733 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970956 5733 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:39.987544 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970959 5733 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970962 5733 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970964 5733 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970967 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970970 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970972 5733 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970975 5733 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970978 5733 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970981 5733 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970984 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970986 5733 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970989 5733 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970991 5733 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970994 5733 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.970996 5733 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971000 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971002 5733 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971005 5733 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971012 5733 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:39.988045 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971014 5733 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971017 5733 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971019 5733 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971022 5733 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971025 5733 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971027 5733 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971030 5733 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971033 5733 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:39.971035 5733 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.971102 5733 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.971998 5733 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972044 5733 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972237 5733 server.go:1014] "Starting client certificate rotation" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972370 5733 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972417 5733 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:24:28 +0000 UTC" deadline="2028-08-13 15:03:20.732350456 +0000 UTC" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972450 5733 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15667h32m40.759902155s" Oct 30 19:30:39.988697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.972644 5733 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:30:39.989203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.977204 5733 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:30:39.989203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.978589 5733 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:30:39.989203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.980544 5733 log.go:25] "Validated CRI v1 image API" Oct 30 19:30:39.989203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.981318 5733 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:30:39.989203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.983524 5733 fs.go:135] Filesystem UUIDs: map[25672f9a-2eab-4922-8e2b-2b8cd1ac713f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e31d558f-d365-425b-8523-a1faabfb0c51:/dev/nvme0n1p3] Oct 30 19:30:39.989412 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.983541 5733 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a/userdata/shm major:0 minor:263 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1/userdata/shm major:0 minor:403 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0/userdata/shm major:0 minor:412 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af7a1cc691ca4595e6bf1f7f8cd9918f3baab302f5b91276c2a5455c5c883883/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af7a1cc691ca4595e6bf1f7f8cd9918f3baab302f5b91276c2a5455c5c883883/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1/userdata/shm major:0 minor:261 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3/userdata/shm major:0 minor:391 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~projected/kube-api-access-8f2qt:{mountpoint:/var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~projected/kube-api-access-8f2qt major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/alertmanager/9 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~empty-dir/config-out major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/kube-api-access-4gtrr:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/kube-api-access-4gtrr major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/tls-assets major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/config-volume major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/web-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29bace5a-fb35-45e9-a2ce-41a61af06524/volumes/kubernetes.io~projected/kube-api-access-g28rx:{mountpoint:/var/lib/kubelet/pods/29bace5a-fb35-45e9-a2ce-41a61af06524/volumes/kubernetes.io~projected/kube-api-access-g28rx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~projected/kube-api-access-j9m54:{mountpoint:/var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~projected/kube-api-access-j9m54 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~secret/metrics-certs major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~projected/kube-api-access-mrtnk:{mountpoint:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~projected/kube-api-access-mrtnk major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46dad1ef-96d8-483e-909b-c082e47acbbd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/46dad1ef-96d8-483e-909b-c082e47acbbd/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56834a9b-e103-4f55-b54f-14e73b4fa1db/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/56834a9b-e103-4f55-b54f-14e73b4fa1db/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/651c14dc-17ea-4608-b927-ba1af3ea8b41/volumes/kubernetes.io~projected/kube-api-access-ff89l:{mountpoint:/var/lib/kubelet/pods/651c14dc-17ea-4608-b927-ba1af3ea8b41/volumes/kubernetes.io~projected/kube-api-access-ff89l major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~projected/kube-api-access-9b6sq:{mountpoint:/var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~projected/kube-api-access-9b6sq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~projected/kube-api-access-9g6vj:{mountpoint:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~projected/kube-api-access-9g6vj major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~projected/kube-api-access-wpc2r:{mountpoint:/var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~projected/kube-api-access-wpc2r major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~projected/kube-api-access-r6s9s:{mountpoint:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~projected/kube-api-access-r6s9s major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~projected/kube-api-access-p2sdb:{mountpoint:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~projected/kube-api-access-p2sdb major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce385369-c46a-4bdd-800a-8ac5ae39a471/volumes/kubernetes.io~projected/kube-api-access-bgkcf:{mountpoint:/var/lib/kubelet/pods/ce385369-c46a-4bdd-800a-8ac5ae39a471/volumes/kubernetes.io~projected/kube-api-access-bgkcf major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/volumes/kubernetes.io~projected/kube-api-access-gqhrb:{mountpoint:/var/lib/kubelet/pods/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/volumes/kubernetes.io~projected/kube-api-access-gqhrb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/config-reloader/2 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/prometheus/15 major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/kube-api-access-pxbkf:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/kube-api-access-pxbkf major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/tls-assets major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/config major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:380 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/web-config major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~projected/kube-api-access-kc22l:{mountpoint:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~projected/kube-api-access-kc22l major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/volumes/kubernetes.io~projected/kube-api-access-g982p:{mountpoint:/var/lib/kubelet/pods/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/volumes/kubernetes.io~projected/kube-api-access-g982p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~projected/kube-api-access-65djh:{mountpoint:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~projected/kube-api-access-65djh major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~projected/kube-api-access-575rj:{mountpoint:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~projected/kube-api-access-575rj major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eed84a66-cd93-4e58-adb3-e097e1b17a14/volumes/kubernetes.io~projected/kube-api-access-nl7xx:{mountpoint:/var/lib/kubelet/pods/eed84a66-cd93-4e58-adb3-e097e1b17a14/volumes/kubernetes.io~projected/kube-api-access-nl7xx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/volumes/kubernetes.io~projected/kube-api-access-c4lvf:{mountpoint:/var/lib/kubelet/pods/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/volumes/kubernetes.io~projected/kube-api-access-c4lvf major:0 minor:61 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e9938068f12be2c0e0a206ea45a1c78cb4d7e51aa4088250fbcbff8cb73c269b/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/642ebdf4876eed26e2ea4b311c2525dad49483112c20ae9191a4863e404dc1c8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b0e64ff7ab40df5f0fc07f126be7937b38a55f62330a7642573ad617e62f163b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9145540bdea9a3c96ee30241738486c54106f1aae15487e0220f7d86de5a3697/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cf40e9a6bd7307229527715a77b3d0e3c20df991e5a8983e670506ef000b7a94/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ca8e5d2befd18673bc235ccfcf73fe9910413d5cb41914c44c63f6bdeeb0a55e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e2db25e4caa5ca969ebdf08215bc1903512941931099b226082c40ec09556a27/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2122de6ab244bc1a9b955dad4b5d3999c77dbb27141c5d688613e34d3070e80f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a1107cd7fd3cd031a34dee21ad2e363c4f865333fc846656b9d69dd6604d2dc6/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/028e24f7f2a4cb3bea8c03fd6ea0f7819bffd2cd7ebab78992de71ed7a1e6846/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/220e1468aed6d6dfdc06972a2f87a1cf70fc1b71f6079b0ca14a6f4375816683/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/541d816d7209c577f8fd0be27f4ed51adc18ca9f3b2e0b1b712d599fe33f5941/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ecd4d1b146cdabf99864fce66465ab78de5befe508499592022f70e42a684d0e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/24dfa8e7e93dd5849fd0e2f0d88636f205220464800d88b5fa833e2900323308/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/16f16e38d632c83f931f5f75f399bf18015371cdf86a673a2fd6912d363182b3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/342d363d463ab87d01f69cca23c0a75b6b0603ac5c33dc80c2bde12419f5e368/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/16d4d0bd740886dc2781d247a1fc5758b93f80bfbd933e0f4fc2e384776aa6ee/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c23bd4b52006b5789aac48a6cc6900b02f5d87c5d6db911c533c5b4859882e3b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/58046d9378d92d0692fb638f1283465480e12dfda98c4117b990f1404a4dd793/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ce0546fdc06499cd220770cd25c8004e4411b263f866bb7fb0243c179af7f0e5/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/93230bf5ce6af9cf80071a11874a3e77f65cfd971cffe170da69ba1b8beb01e5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f16b7c5e54c3e7db99446354c13be300b18585a9f9b5ef7b9a2909c439143da7/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/67423489c62aa836320c31931f5857c362db06e51a2311dc240b2e25d97d27bf/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/9c3b25edb36f653173eb3aefbe77c389a63b47945598ba5bdc5801f58cee0cd2/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/04edfee9b803991c4f7fa145ca5cefba7c4f1a3e5c9a53868ab1299cc371ab65/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d443fdcb8b6dd63985648f5576839be11b38b50ce827557df31af41b59d563a3/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e72402eb7c1275e12d2717e37c4836feed966dd897ced9988b246554c4f1a0b1/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/7425028b4f73ba015378ccf71c0af98d8bd446a5103581e3459a7b38d6fb0caa/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/797aee25ebb624bc0831e870ecb1320ff691dc45227d4b6a2763be947fcab7c9/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/028e444c8b45a0b21f26608e574cf36bcafd21641f18b49769a098bd7c4e1f00/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/c66c2ac0aa1ee75b8d5d8440dbb83089d0557ba11738bd65ab09188f6a18f76f/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/b93361ea7076da060add0090c94d52517e3bbe83f2f6ce7893b836f23e0dd49b/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/3805159434d64aff1ea46c7646d7f4497eef42c0ddb63ef15e7786f0b9f14a66/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/0535997866c26a64d78f4aecf2211c32465b6ea7763785688725dbe8ed21ce7d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/e48097ac7b2a8f4e5925d863edea078f2230a1067bdcb874f6e052c7fe2e867e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/f49add07ffcd788d6cc26b4d6914193eeea91d10bb273ee9d4c7516efb27ee51/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/2bf5394c4070c85769f116a50902a4f201b5469d39c7eb6d03ed7a0463a874a3/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/a68d3c98ac0545216bc7ce2bb5ba92169db377dea2c905b3068483d99e701654/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/319054daf94c26aa0b7c451fb41d5fee89f0fb34516561a7de3f8d956d2a296c/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/908bebbf4525c9930efdb8661dfee92cda36ce506daa81da5204fb2a9670873d/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/e01fb8fb6bc2017ad854a78689b2d7e29983c5e8b5aa463c5011912a97725ee4/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/33ab0a87f689f1542b5e07726c4ec2196ad1236851e11025239b84b2fea10b4c/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/22f4d272a2727972735f5afe7e9ef8fabbecf59ec80c980ac730c77de81ad021/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/6a8d31fdc52a72b2ee9dc8cd45763110e7de98194d9e839b889c651be3f6e6f2/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/6b690a313bb3735c6c8343c244612f35e027775c078ee8cdc446b71a8b214952/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/07a162e3406c891263cde561787638647289b158f67352ab5c8c2d66911d7c9d/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/f349cf6f324653eab885fb983cd49770e74cf1be7ac2d50a02e532e55259f3a6/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/6dd8945907f2453b3c7858ed165e1a974a8e76c8ea2900508c1ff444ab96d9f3/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/b9d1191eade8aa360db0852f8b38c45b6ebc1bfe5c5259e84c2595bf521b1391/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/52119e81a476188e88cd335a0b9922163abb886ca36a9aa48fa4d31f3ab1211d/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/057bafb027ce705bbc88b584aa2edbe6b6f944547737a4bce0487c95279443eb/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/809e505e260ebe8b625bffd3a3b63b579377ecb8322f490f7e01df9eca93c746/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-395:{mountpoint:/var/lib/containers/storage/overlay/d3ce1ac7075659bd7bf12b1f5e42d4c406a46bfb58373e47775b9eb4cf729185/merged major:0 minor:395 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/dc5aa501600901a829c11688dd3826eff3948fa2f7b77ed5c9dee79c0ae98f27/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/308bb5534f69324cbf4475e294f5ccd4c747f2fd44b3ba16f26ed9b184368023/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/d62173e5fb1d530076fd55d3d2532736d91db4a56d995a7d3ca077045456b309/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/bab352d4206c8774c00107f9de89c5035334cdd9c0829ba5787ad879013127a2/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/31ff06916086290a7cc0ddbc7f33817d712d0fd7b702b1eb4ad35c3a1bbba545/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/68f42a823e90de6279e2431193e125c26b50ef95134aba41ef57e1e6aa743077/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/9528dfa863d8f54c349e8abeb5199826cacca04ca44bff318f2114da1a6788f5/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/9b22b6bee3d9c5c69f7dfa615403b809e6ce993d9142c58ac802b85617c65992/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/2380fd3928200dcc87321c4060df487719148ca9e0e0a798fa258b75fa69e32a/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/43907901483bc30bc425e22ec69a5e118eee287e990088ec90bce4d54a6d9ed5/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/cd1a10e9605ab5837f767e861e811cfc2d0478232a91bd67cbcd0407dcd12037/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/c7e154b51cea889b993c9b5d372cf163100c9296f3c1ce940d92987bd35f851a/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/56d01027ec2148c3bf775f0b040c353da383e8c445226b60b29350821e745241/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/2beae6ad824d2ea25277f19427753fe664c51d68b1390c55ef1fd60f37e7d54e/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/049c616bfab68e445e103567649577c19bc62870d3e1cee3e16b4095224f248e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a840a5cd6aa0fb5297fb88157c11f32ddd501bf0a18cae48dab26fc4aa372220/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a008d2b8553ebc9fcfd70c6510406fe90e28af77eb8c5949f1aeec4c179034da/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/410ee0706286065147fd023256046ef0f4917b578a8f4159f08ca825ecb7e919/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3430cea64773c3262487b1476a4760552b75eb1f401b028c4cb866e5af535418/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e1b50971c3439a04c0a752aff39f656de004f6e928f922e338f9fed13eef4c62/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7866f36e16259595689f8e0902c4086d6de10de69abfc82a37b61715509a3a11/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/dc2841edc08d4e48d7a02ff013b6535321cfe299e24e7fb9ea347fbaad4b6f10/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/67f773d397465f312bccf9663f44c37a2a7a02f047c6c12113c053f86d2bb207/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:30:39.992230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.991814 5733 manager.go:217] Machine: {Timestamp:2026-10-30 19:30:39.991349056 +0000 UTC m=+0.059184616 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100127 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a9e0d17a42c5d1f72965d2b035f7d SystemUUID:ec2a9e0d-17a4-2c5d-1f72-965d2b035f7d BootID:0f5b541f-697e-49e8-ab76-f8fc80da20c0 Filesystems:[{Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af7a1cc691ca4595e6bf1f7f8cd9918f3baab302f5b91276c2a5455c5c883883/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc4132d0a5dfeb3dbd313e65802b2ee31e36b96a93ca0cbdd90d12e926d1b19e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~projected/kube-api-access-65djh DeviceMajor:0 DeviceMinor:257 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1922d5017d3f19becddea02363a3b7499a64384acd1828f4e3d83724b0341fc1/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56834a9b-e103-4f55-b54f-14e73b4fa1db/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:401 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/volumes/kubernetes.io~projected/kube-api-access-gqhrb DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:384 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/46dad1ef-96d8-483e-909b-c082e47acbbd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4c66fdaa640b9a37c1cd9f429bdce1f34221eb3ce92a3722fc108a4e656296a/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:388 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~projected/kube-api-access-j9m54 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/467871da2ee36d684e89f0c6764d4727cc71f2bc1824d2608f94cd873f7b7b63/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~projected/kube-api-access-p2sdb DeviceMajor:0 DeviceMinor:322 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/339d4aa67a827a6ede288e68ca5a031dbefa31bde90071a5df965ce8cb05f85a/userdata/shm DeviceMajor:0 DeviceMinor:263 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9738ce668e8673d3888efb4804f3e450f3010981e4bed1397ff32cf8ed0e42ec/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~projected/kube-api-access-kc22l DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eed84a66-cd93-4e58-adb3-e097e1b17a14/volumes/kubernetes.io~projected/kube-api-access-nl7xx DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77bbd69f8ceee2dcfb4d1279173c7f8773b76d6d43a2250a71af1c4eacf77d91/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:382 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~projected/kube-api-access-9b6sq DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33381540-cec9-4797-a8b9-be05dd1f13c0/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:402 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ec720c191542f7daa893687967d851cabe5e602c22b8ef735c01991cfafd6f1/userdata/shm DeviceMajor:0 DeviceMinor:403 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~projected/kube-api-access-575rj DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/305e31a3a8a0e38c1a4d09d1aaad7763ee874317445417330ddc98466b5318e7/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:319 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02819071ba4030005f899d5537208bdd887b910ef0ab0207524fad29b67164d6/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce385369-c46a-4bdd-800a-8ac5ae39a471/volumes/kubernetes.io~projected/kube-api-access-bgkcf DeviceMajor:0 DeviceMinor:407 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~projected/kube-api-access-mrtnk DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:381 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d87433f0-7272-49cb-a84e-20565709bdae/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:252 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/38afb684-e999-4c10-9281-62f633431cc6/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c8765ee282f991e592cad56288d274f40a980793088f89657a6c90272b2f21c2/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:383 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:389 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/volumes/kubernetes.io~projected/kube-api-access-c4lvf DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71be183682bce887696f8cb24f2966a791aacb619dd7653001cbcbe7d9569755/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/590d469cc8e8e4ebccffb4d286d2839d924d0e16f2b16cf55a77555421b62983/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29bace5a-fb35-45e9-a2ce-41a61af06524/volumes/kubernetes.io~projected/kube-api-access-g28rx DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/72fe1029-ecc8-4b99-9387-659ae6e414b3/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:376 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:380 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:385 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/kube-api-access-pxbkf DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-395 DeviceMajor:0 DeviceMinor:395 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/651c14dc-17ea-4608-b927-ba1af3ea8b41/volumes/kubernetes.io~projected/kube-api-access-ff89l DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d903d7930170b6fc21230866f0e29318c1195eaa00c0e054bafee8909ee2813/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:287 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b36aff52-af76-477f-990d-9900e4c43954/volumes/kubernetes.io~projected/kube-api-access-r6s9s DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:286 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0678842b-b564-4b68-ae5b-aaffd1680479/volumes/kubernetes.io~projected/kube-api-access-8f2qt DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1c7a804-0740-4d51-884b-ea661daf80be/volumes/kubernetes.io~projected/kube-api-access-9g6vj DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3/userdata/shm DeviceMajor:0 DeviceMinor:391 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b76e60-de88-4c65-b570-fb4772dfbb5b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/volumes/kubernetes.io~projected/kube-api-access-g982p DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd50ccf3f234b193f02bfa44fe4473156b387d09d61455dbab3fd78b088840c1/userdata/shm DeviceMajor:0 DeviceMinor:261 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b107b211-0176-4170-8d0f-85d4b36a4f30/volumes/kubernetes.io~projected/kube-api-access-wpc2r DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b63544cccd8be19ce70bcce8d2c198693252159cb9335e72c48efc75aac491c3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes/kubernetes.io~projected/kube-api-access-4gtrr DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:320 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:387 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8f88bfd6c04a5299f681cd777b4ecc70170f0e8bc86bf7657f7ebf302520bf0/userdata/shm DeviceMajor:0 DeviceMinor:412 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1922d5017d3f19b MacAddress:8a:39:f3:5c:ee:1b Speed:10000 Mtu:8901} {Name:590d469cc8e8e4e MacAddress:86:e8:e7:dd:64:33 Speed:10000 Mtu:8901} {Name:77bbd69f8ceee2d MacAddress:d2:f3:6e:f6:75:26 Speed:10000 Mtu:8901} {Name:78075e7633b90b0 MacAddress:e6:26:f1:9a:98:19 Speed:10000 Mtu:8901} {Name:8ec720c191542f7 MacAddress:e6:3c:26:19:a3:af Speed:10000 Mtu:8901} {Name:9738ce668e8673d MacAddress:96:16:0d:4e:3a:d6 Speed:10000 Mtu:8901} {Name:a8f88bfd6c04a52 MacAddress:de:c4:cf:0e:3b:8a Speed:10000 Mtu:8901} {Name:b63544cccd8be19 MacAddress:4a:99:02:1c:c3:ed Speed:10000 Mtu:8901} {Name:bc0468473d01407 MacAddress:aa:60:e9:95:7b:07 Speed:10000 Mtu:8901} {Name:bd50ccf3f234b19 MacAddress:72:78:bc:f5:0b:20 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e6:ca:a0:61 Speed:0 Mtu:9001} {Name:br-int MacAddress:26:f1:11:38:bc:75 Speed:0 Mtu:8901} {Name:c8765ee282f991e MacAddress:6a:7b:b7:6d:4a:1a Speed:10000 Mtu:8901} {Name:d52206c1af0e59e MacAddress:7e:37:1d:46:d0:c3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e6:ca:a0:61 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:8e:ce:85:e1:8b:04 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:4b:03:3c:49:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:30:39.992485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992234 5733 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:30:39.992485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992278 5733 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:30:39.992558 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992542 5733 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:30:39.992684 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992561 5733 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-103.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:30:39.992727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992686 5733 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:30:39.992727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992693 5733 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:30:39.992727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992702 5733 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:30:39.992727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992725 5733 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:30:39.992825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992797 5733 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:30:39.992963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.992954 5733 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:30:39.993094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993082 5733 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:30:39.993133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993125 5733 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:30:39.993162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993137 5733 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:30:39.993162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993146 5733 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:30:39.993162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993156 5733 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:30:39.993244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.993168 5733 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:30:39.995617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.995600 5733 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:30:39.995617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.995615 5733 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:30:39.996051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.996026 5733 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:30:39.998743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.998728 5733 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:30:39.998743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.998744 5733 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:30:39.998974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.998836 5733 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:30:39.999030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999018 5733 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:30:39.999076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999048 5733 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:30:39.999168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999159 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:30:39.999211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999174 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:30:39.999211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999180 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:30:39.999211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999185 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:30:39.999211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999190 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:30:39.999211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999196 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999216 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999223 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999229 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999235 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999243 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999252 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999277 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:30:39.999358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:39.999282 5733 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:30:40.002655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.002641 5733 server.go:1263] "Started kubelet" Oct 30 19:30:40.002744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.002717 5733 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:30:40.002798 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.002734 5733 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:30:40.002844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.002811 5733 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:30:40.003159 ip-10-0-139-103 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:30:40.003288 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.003261 5733 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:30:40.004613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.004590 5733 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:30:40.008999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.007236 5733 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:30:40.009114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.009040 5733 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.009558 5733 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-103.us-east-2.compute.internal\" not found" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.010892 5733 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.010903 5733 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.011231 5733 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.011624 5733 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:24:30 +0000 UTC" deadline="2028-05-21 12:58:25.171144762 +0000 UTC" Oct 30 19:30:40.012120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.011662 5733 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13649h27m45.159486541s" Oct 30 19:30:40.014562 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.014539 5733 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:30:40.014562 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.014559 5733 factory.go:55] Registering systemd factory Oct 30 19:30:40.014562 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.014567 5733 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:30:40.015421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015361 5733 factory.go:153] Registering CRI-O factory Oct 30 19:30:40.015421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015379 5733 factory.go:223] Registration of the crio container factory successfully Oct 30 19:30:40.015421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015404 5733 factory.go:103] Registering Raw factory Oct 30 19:30:40.015421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015418 5733 manager.go:1199] Started watching for new ooms in manager Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015407 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36aff52-af76-477f-990d-9900e4c43954" volumeName="kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015464 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015482 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015501 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015513 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0" volumeName="kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015526 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015543 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015564 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651c14dc-17ea-4608-b927-ba1af3ea8b41" volumeName="kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015584 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36aff52-af76-477f-990d-9900e4c43954" volumeName="kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015596 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015612 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eed84a66-cd93-4e58-adb3-e097e1b17a14" volumeName="kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015663 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eed84a66-cd93-4e58-adb3-e097e1b17a14" volumeName="kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015694 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce" volumeName="kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015849 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36aff52-af76-477f-990d-9900e4c43954" volumeName="kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015902 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0678842b-b564-4b68-ae5b-aaffd1680479" volumeName="kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015922 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015928 5733 manager.go:319] Starting recovery of all containers Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015945 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72fe1029-ecc8-4b99-9387-659ae6e414b3" volumeName="kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015960 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015979 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.015993 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016006 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016025 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016040 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72fe1029-ecc8-4b99-9387-659ae6e414b3" volumeName="kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016381 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016396 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36aff52-af76-477f-990d-9900e4c43954" volumeName="kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016416 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016431 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce67c2e9-3fdd-4a2b-b627-70993c44ea9f" volumeName="kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016443 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016473 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016493 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0678842b-b564-4b68-ae5b-aaffd1680479" volumeName="kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016507 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651c14dc-17ea-4608-b927-ba1af3ea8b41" volumeName="kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016529 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016553 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016566 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016584 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b76e60-de88-4c65-b570-fb4772dfbb5b" volumeName="kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016598 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce" volumeName="kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016616 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016630 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016644 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016664 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce67c2e9-3fdd-4a2b-b627-70993c44ea9f" volumeName="kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016677 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016695 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016723 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016741 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016762 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016776 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016793 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016807 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016820 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016839 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b76e60-de88-4c65-b570-fb4772dfbb5b" volumeName="kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016850 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29bace5a-fb35-45e9-a2ce-41a61af06524" volumeName="kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016861 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33381540-cec9-4797-a8b9-be05dd1f13c0" volumeName="kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016871 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b107b211-0176-4170-8d0f-85d4b36a4f30" volumeName="kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016887 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce385369-c46a-4bdd-800a-8ac5ae39a471" volumeName="kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016905 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016919 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651c14dc-17ea-4608-b927-ba1af3ea8b41" volumeName="kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016950 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016965 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b107b211-0176-4170-8d0f-85d4b36a4f30" volumeName="kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016980 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b107b211-0176-4170-8d0f-85d4b36a4f30" volumeName="kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.016998 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.017013 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b76e60-de88-4c65-b570-fb4772dfbb5b" volumeName="kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.017031 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.017052 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config" seLinuxMountContext="" Oct 30 19:30:40.017277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.017077 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.017096 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="651c14dc-17ea-4608-b927-ba1af3ea8b41" volumeName="kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021125 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72fe1029-ecc8-4b99-9387-659ae6e414b3" volumeName="kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021152 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021167 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021182 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021199 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021214 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46dad1ef-96d8-483e-909b-c082e47acbbd" volumeName="kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021228 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021247 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021260 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021275 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021292 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021306 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46dad1ef-96d8-483e-909b-c082e47acbbd" volumeName="kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.019995 5733 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.020992 5733 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021320 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b107b211-0176-4170-8d0f-85d4b36a4f30" volumeName="kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume" seLinuxMountContext="" Oct 30 19:30:40.021369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021371 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021389 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021402 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021416 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021430 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38afb684-e999-4c10-9281-62f633431cc6" volumeName="kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021441 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d6376030-0944-400b-942f-286dc1cce30b" volumeName="kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021455 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021466 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d87433f0-7272-49cb-a84e-20565709bdae" volumeName="kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021477 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0" volumeName="kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021488 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0" volumeName="kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021503 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29bace5a-fb35-45e9-a2ce-41a61af06524" volumeName="kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021514 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6539c7b-9167-4d62-a086-6c02a73827d2" volumeName="kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021530 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021543 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33381540-cec9-4797-a8b9-be05dd1f13c0" volumeName="kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021555 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56834a9b-e103-4f55-b54f-14e73b4fa1db" volumeName="kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021568 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eed84a66-cd93-4e58-adb3-e097e1b17a14" volumeName="kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021580 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021591 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021605 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22cf1b54-38ba-446d-8083-b66ba618a7db" volumeName="kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021616 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" volumeName="kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021629 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0" volumeName="kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021641 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72fe1029-ecc8-4b99-9387-659ae6e414b3" volumeName="kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021653 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1c7a804-0740-4d51-884b-ea661daf80be" volumeName="kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021665 5733 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b36aff52-af76-477f-990d-9900e4c43954" volumeName="kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides" seLinuxMountContext="" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021676 5733 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:30:40.022613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.021684 5733 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:30:40.028321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.026557 5733 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.028321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.026623 5733 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:30:40.040509 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.040461 5733 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:30:40.041841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.041816 5733 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:30:40.041930 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.041845 5733 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:30:40.041930 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.041873 5733 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:30:40.042002 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.041922 5733 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:30:40.046966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.046946 5733 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.054964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.054940 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d" exitCode=0 Oct 30 19:30:40.058755 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.058721 5733 generic.go:356] "Generic (PLEG): container finished" podID="a1c7a804-0740-4d51-884b-ea661daf80be" containerID="f83f180862c3048f480a2ef67ed067b08572e380d22c880b1752516dbdae3304" exitCode=0 Oct 30 19:30:40.072046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.072024 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" exitCode=0 Oct 30 19:30:40.082714 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082689 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="8bbd1e2709c1f081d0a7f50849013179d388deed72ac245aba44be6a112bcc9e" exitCode=0 Oct 30 19:30:40.082714 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082712 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="8bd1f31e71ee20c09e5b5711a172ecc220a8248da68857c12e72efe3d7b7c601" exitCode=0 Oct 30 19:30:40.082887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082719 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="7f59bbcca1cdb218bd1ef367cab3b09eb35eb96edf77346a3bb38e1dcec7d858" exitCode=0 Oct 30 19:30:40.082887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082728 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="30739ea3520f26c0b57c7d495296a68e2e038b1d212036aeb6424ef366d1006f" exitCode=0 Oct 30 19:30:40.082887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082737 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="d5eef995edce3684d810de402538ed3d5a3a237f158c291ea4d832418dd9e451" exitCode=0 Oct 30 19:30:40.082887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.082743 5733 generic.go:356] "Generic (PLEG): container finished" podID="651c14dc-17ea-4608-b927-ba1af3ea8b41" containerID="cbfa5d3ba915e9089abeffaeef805886838dc7d5060a2e7395a594137392978d" exitCode=0 Oct 30 19:30:40.084596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.084571 5733 generic.go:356] "Generic (PLEG): container finished" podID="f224810f96f29df31a3e8747164653d1" containerID="680303dc2b2394ba09904382e111d2b816b6236954b8c0b2fa5272d3e4e32c1e" exitCode=0 Oct 30 19:30:40.133694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.133639 5733 manager.go:324] Recovery completed Oct 30 19:30:40.143221 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.143194 5733 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:30:40.156779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156757 5733 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:30:40.156779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156772 5733 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156788 5733 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156900 5733 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156911 5733 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156928 5733 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156934 5733 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156939 5733 policy_none.go:50] "Start" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156948 5733 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:30:40.156951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.156957 5733 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:30:40.157238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.157092 5733 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:30:40.157238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.157100 5733 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:30:40.157238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.157106 5733 policy_none.go:44] "Start" Oct 30 19:30:40.161640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.161610 5733 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:30:40.164164 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.164150 5733 manager.go:341] "Starting Device Plugin manager" Oct 30 19:30:40.164707 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.164179 5733 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:30:40.164760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.164710 5733 server.go:87] "Starting device plugin registration server" Oct 30 19:30:40.164956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.164947 5733 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:30:40.164986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.164957 5733 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:30:40.165101 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.165077 5733 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:30:40.165230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.165190 5733 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:30:40.165293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.165240 5733 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:30:40.165293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.165247 5733 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:30:40.165442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.165287 5733 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:30:40.165226711Z","UUID":"867d0dc3-f715-4fba-ae34-e967286bb78f","Handler":null,"Name":"","Endpoint":""} Oct 30 19:30:40.166714 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.166692 5733 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:30:40.166821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.166721 5733 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:30:40.166875 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.166842 5733 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:30:40.265556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.265521 5733 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:40.267420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.267402 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:40.267540 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.267436 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:40.267540 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.267450 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:40.267540 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.267509 5733 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.283470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.283449 5733 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.283539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.283512 5733 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.344252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344194 5733 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal"] Oct 30 19:30:40.344784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344743 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"e4925bc4d897d213bab4d8b38932653e","Type":"ContainerStarted","Data":"fe38c22a051f7f3ef34b1f316e373f73cfc0cdd4b74ac8a1f59c58c9bc4ffa85"} Oct 30 19:30:40.344886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344873 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerStarted","Data":"5db72b8e3365cd2a84d9df19edbca5a4d2bb74d11260134d6f502e3457d04dec"} Oct 30 19:30:40.344957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344893 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerDied","Data":"680303dc2b2394ba09904382e111d2b816b6236954b8c0b2fa5272d3e4e32c1e"} Oct 30 19:30:40.344957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344906 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" event={"ID":"f224810f96f29df31a3e8747164653d1","Type":"ContainerStarted","Data":"2df9f49a9970b76fd6039aa9df90fbcdb8e3fa118dda62747ce98b530b878c81"} Oct 30 19:30:40.345070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.344975 5733 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.345121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.345073 5733 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.361163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.361144 5733 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:40.361244 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.361189 5733 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.361854 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.361844 5733 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:40.361894 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:40.361867 5733 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.428768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.428683 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.428768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.428723 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.428768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.428750 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529205 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529252 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529268 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529309 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529318 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e4925bc4d897d213bab4d8b38932653e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"e4925bc4d897d213bab4d8b38932653e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.529442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.529327 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f224810f96f29df31a3e8747164653d1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" (UID: \"f224810f96f29df31a3e8747164653d1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:40.993913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:40.993868 5733 apiserver.go:52] "Watching apiserver" Oct 30 19:30:41.011429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.011402 5733 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:30:41.012041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.012017 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bd8695446-nbt67","openshift-multus/multus-2hcjh","openshift-ovn-kubernetes/ovnkube-node-h6kbw","openshift-console/console-6864546c55-lv6zs","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf","openshift-multus/network-metrics-daemon-8nnkd","openshift-image-registry/node-ca-7gh2g","openshift-ingress-canary/ingress-canary-bct4d","kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal","openshift-dns/dns-default-bjt85","openshift-monitoring/node-exporter-tx4kh","openshift-network-operator/iptables-alerter-2pmgc","openshift-network-diagnostics/network-check-target-mrpl8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn","openshift-insights/insights-runtime-extractor-nrp79","openshift-multus/multus-additional-cni-plugins-l89fd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-2qlwk","openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g","kube-system/konnectivity-agent-kms8d","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-jllfr","openshift-dns/node-resolver-jcmd8"] Oct 30 19:30:41.021203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021169 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\"" Oct 30 19:30:41.021203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021160 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\"" Oct 30 19:30:41.021394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021290 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:30:41.021394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021371 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:30:41.021503 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021395 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\"" Oct 30 19:30:41.021834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021810 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.021834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021826 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.021978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021855 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:30:41.021978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021855 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.021978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.021883 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.022745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.022725 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:30:41.022745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.022736 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.023150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023126 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:30:41.023246 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023168 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\"" Oct 30 19:30:41.023246 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023225 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.023324 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023265 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\"" Oct 30 19:30:41.023324 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023269 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\"" Oct 30 19:30:41.023324 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023309 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:30:41.023562 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023540 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.023638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023542 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:30:41.023638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.023595 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.024440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024417 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.024532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024456 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.024532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024492 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.024532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024524 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:30:41.024696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024577 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\"" Oct 30 19:30:41.024696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024691 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\"" Oct 30 19:30:41.024784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.024779 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:30:41.027841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.027824 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:30:41.029243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029219 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.029368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029264 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:30:41.029368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029300 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.029457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029369 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.029457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029410 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.029457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029424 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:30:41.029658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029645 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.029709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.029688 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.030322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.030303 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:30:41.030402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.030358 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:30:41.030554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.030539 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.030613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.030574 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.031342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031320 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.031419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031355 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:30:41.031419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031359 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:30:41.031419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031361 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:30:41.031419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031401 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:30:41.031596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031561 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:30:41.031739 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.031718 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:30:41.032860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.032840 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:30:41.032860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.032847 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\"" Oct 30 19:30:41.033021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.032876 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:30:41.033021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.032912 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qv2b2\"" Oct 30 19:30:41.033329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033315 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\"" Oct 30 19:30:41.033385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033329 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033597 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033610 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033611 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033627 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pr8mc\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033631 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\"" Oct 30 19:30:41.033671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033659 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 19:30:41.033998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033680 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 19:30:41.033998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.033685 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:30:41.034904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.034892 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:30:41.035494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035476 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:30:41.035548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035507 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\"" Oct 30 19:30:41.035548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035534 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:30:41.035548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035541 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035544 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035545 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035569 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035589 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035622 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 19:30:41.035685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035645 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wsp2p\"" Oct 30 19:30:41.035956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035698 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:30:41.035956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035708 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\"" Oct 30 19:30:41.035956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.035770 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jqkdh\"" Oct 30 19:30:41.105084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.105043 5733 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:41.105214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.105140 5733 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:41.114698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.114678 5733 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:30:41.120125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.120100 5733 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:41.120203 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.120159 5733 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:41.120484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.120465 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 19:30:41.120639 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.120620 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:30:41.120690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.120653 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:30:41.120985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.120967 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:30:41.121165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.121152 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:30:41.121202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.121190 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 19:30:41.121202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.121197 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:30:41.125559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.125542 5733 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:41.125645 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.125629 5733 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal" Oct 30 19:30:41.127040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.127022 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:30:41.128086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.128051 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:30:41.128185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.128092 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:30:41.128185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.128096 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:30:41.128185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.128127 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 19:30:41.128185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.128175 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:30:41.130335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130307 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8apo69g0vql64\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130310 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130358 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130375 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5kn8\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130387 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130393 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130395 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:30:41.130486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130392 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:30:41.130762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130568 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 19:30:41.130762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130597 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6b35nveaduvtq\"" Oct 30 19:30:41.130762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130720 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.130762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130745 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.130762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130761 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.131014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130777 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.131014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130794 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.131014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130844 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.131014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130891 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.131014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.130938 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131029 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131033 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131115 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131170 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-client-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131172 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131223 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131227 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.131285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131271 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131299 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131393 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131412 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131427 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131459 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131481 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131502 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131506 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131518 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpc2r\" (UniqueName: \"kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131537 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131552 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131602 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.131668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131638 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131677 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131681 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-serviceca\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131733 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131785 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131822 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-cni-binary-copy\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131831 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9m54\" (UniqueName: \"kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131916 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131936 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.131978 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132052 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.132196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132172 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132204 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-client-certs\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132269 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132315 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132345 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132367 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/38afb684-e999-4c10-9281-62f633431cc6-secret-metrics-server-tls\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132372 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-env-overrides\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132410 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132448 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132509 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132540 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132555 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132560 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-config\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132572 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132527 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132605 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132660 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132697 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46dad1ef-96d8-483e-909b-c082e47acbbd-agent-certs\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132705 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132737 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132753 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtnk\" (UniqueName: \"kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132759 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132792 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132796 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b36aff52-af76-477f-990d-9900e4c43954-ovnkube-script-lib\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132824 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.132936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132918 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33381540-cec9-4797-a8b9-be05dd1f13c0-metrics-certs\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132960 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132971 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lvf\" (UniqueName: \"kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.132978 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133003 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133047 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133140 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133165 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133233 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133305 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133458 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133546 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133583 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133604 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133619 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133645 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28rx\" (UniqueName: \"kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133653 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133662 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133705 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133753 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc22l\" (UniqueName: \"kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133788 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133818 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133823 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133842 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133866 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133894 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133919 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133949 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29bace5a-fb35-45e9-a2ce-41a61af06524-iptables-alerter-script\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.133964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133968 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.133996 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134023 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134052 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134113 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134166 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134208 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134235 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134261 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134294 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134315 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-accelerators-collector-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134319 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134412 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-daemon-config\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134473 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134503 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134508 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134539 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134555 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134565 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134574 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-tls\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134595 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.134669 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134690 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134699 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g982p\" (UniqueName: \"kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.134757 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.634731518 +0000 UTC m=+1.702567084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:41.134821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134762 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d87433f0-7272-49cb-a84e-20565709bdae-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134867 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0678842b-b564-4b68-ae5b-aaffd1680479-cert\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134866 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134922 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b107b211-0176-4170-8d0f-85d4b36a4f30-tmp-dir\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134929 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d87433f0-7272-49cb-a84e-20565709bdae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.134945 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135018 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-textfile\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135052 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135116 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.135159 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135168 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135252 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.135274 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.635262697 +0000 UTC m=+1.703098269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135297 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135321 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135337 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135367 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135384 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135397 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/72fe1029-ecc8-4b99-9387-659ae6e414b3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135402 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff89l\" (UniqueName: \"kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135460 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135479 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135496 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135512 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135527 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135543 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b36aff52-af76-477f-990d-9900e4c43954-ovn-node-metrics-cert\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135555 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135604 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.135629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135627 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135646 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135662 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135679 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135694 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135711 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135726 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135745 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.135773 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7xx\" (UniqueName: \"kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136380 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.136449 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.136510 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.636495074 +0000 UTC m=+1.704330635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136599 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-binary-copy\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.136721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136708 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136777 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136818 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f2qt\" (UniqueName: \"kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136840 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-metrics-server-audit-profiles\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136888 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.136943 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137008 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137203 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.137368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137315 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-tuned\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.137747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137269 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6sq\" (UniqueName: \"kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.137747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137573 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.137890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137814 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/56834a9b-e103-4f55-b54f-14e73b4fa1db-original-pull-secret\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.137890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137878 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8b76e60-de88-4c65-b570-fb4772dfbb5b-tmp\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.137983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137954 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-tmp-dir\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.138030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.137980 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46dad1ef-96d8-483e-909b-c082e47acbbd-konnectivity-ca\") pod \"konnectivity-agent-kms8d\" (UID: \"46dad1ef-96d8-483e-909b-c082e47acbbd\") " pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:30:41.138102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138033 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/72fe1029-ecc8-4b99-9387-659ae6e414b3-data-volume\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.138152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138100 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.138152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138137 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.138239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138153 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.138239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138167 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.138239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138215 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.138373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138248 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6vj\" (UniqueName: \"kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.138373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138292 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.138495 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138473 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.138550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138532 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.138604 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138578 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.138651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138613 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.138695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138658 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.138788 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138770 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.138860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138795 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.138983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138966 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b107b211-0176-4170-8d0f-85d4b36a4f30-config-volume\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.139029 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.138973 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.139136 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.139075 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:41.139178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139164 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.139245 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.139232 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.639215347 +0000 UTC m=+1.707050907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:41.139340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139243 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/38afb684-e999-4c10-9281-62f633431cc6-audit-log\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.139340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139298 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.139433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139358 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:41.139492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139474 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.139731 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139710 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.139780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139719 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqhrb\" (UniqueName: \"kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.139849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139774 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.139916 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139897 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.139962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139943 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.140010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.139990 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.140122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140024 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.140122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140112 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.140210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140147 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.140210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140177 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.140210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140192 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/651c14dc-17ea-4608-b927-ba1af3ea8b41-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.140390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140238 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.140462 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140444 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38afb684-e999-4c10-9281-62f633431cc6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.140508 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140495 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65djh\" (UniqueName: \"kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.140559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140529 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.140607 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140563 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.140607 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140593 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.140693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140650 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.140693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140681 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.140779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140710 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.140779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140727 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1c7a804-0740-4d51-884b-ea661daf80be-metrics-client-ca\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.140779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140738 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.140904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140892 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.140948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140923 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.140998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140956 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.140998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.140989 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.141107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141032 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.141154 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.141127 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:41.141226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141209 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.141505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141487 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.141563 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141540 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.141611 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141580 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.141660 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141636 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.141733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141667 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.141788 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.141744 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.141835 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.141793 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:41.141885 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.141842 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.641827634 +0000 UTC m=+1.709663193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:41.141971 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.141938 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.641920022 +0000 UTC m=+1.709755583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:41.142185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142165 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.142307 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142290 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.142307 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142291 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.142402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142338 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b107b211-0176-4170-8d0f-85d4b36a4f30-metrics-tls\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.142486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142469 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575rj\" (UniqueName: \"kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.142530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142480 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.142580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142522 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.142580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142527 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.142580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142567 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.142736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142614 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6s9s\" (UniqueName: \"kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.142736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142651 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.142990 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.142970 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.143037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.143012 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.144186 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.144166 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpc2r\" (UniqueName: \"kubernetes.io/projected/b107b211-0176-4170-8d0f-85d4b36a4f30-kube-api-access-wpc2r\") pod \"dns-default-bjt85\" (UID: \"b107b211-0176-4170-8d0f-85d4b36a4f30\") " pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:41.144332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.144314 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9m54\" (UniqueName: \"kubernetes.io/projected/33381540-cec9-4797-a8b9-be05dd1f13c0-kube-api-access-j9m54\") pod \"network-metrics-daemon-8nnkd\" (UID: \"33381540-cec9-4797-a8b9-be05dd1f13c0\") " pod="openshift-multus/network-metrics-daemon-8nnkd" Oct 30 19:30:41.146269 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.146252 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lvf\" (UniqueName: \"kubernetes.io/projected/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-kube-api-access-c4lvf\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.146406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.146391 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"prometheus-k8s-0\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:41.149645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.149626 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc22l\" (UniqueName: \"kubernetes.io/projected/d87433f0-7272-49cb-a84e-20565709bdae-kube-api-access-kc22l\") pod \"kube-state-metrics-5dc8d7d89b-fx95g\" (UID: \"d87433f0-7272-49cb-a84e-20565709bdae\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" Oct 30 19:30:41.149709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.149698 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28rx\" (UniqueName: \"kubernetes.io/projected/29bace5a-fb35-45e9-a2ce-41a61af06524-kube-api-access-g28rx\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.150320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.150299 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"alertmanager-main-0\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:41.150546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.150533 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtnk\" (UniqueName: \"kubernetes.io/projected/38afb684-e999-4c10-9281-62f633431cc6-kube-api-access-mrtnk\") pod \"metrics-server-7bd8695446-nbt67\" (UID: \"38afb684-e999-4c10-9281-62f633431cc6\") " pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.153400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.153383 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkcf\" (UniqueName: \"kubernetes.io/projected/ce385369-c46a-4bdd-800a-8ac5ae39a471-kube-api-access-bgkcf\") pod \"network-check-target-mrpl8\" (UID: \"ce385369-c46a-4bdd-800a-8ac5ae39a471\") " pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:41.155267 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.155245 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f2qt\" (UniqueName: \"kubernetes.io/projected/0678842b-b564-4b68-ae5b-aaffd1680479-kube-api-access-8f2qt\") pod \"ingress-canary-bct4d\" (UID: \"0678842b-b564-4b68-ae5b-aaffd1680479\") " pod="openshift-ingress-canary/ingress-canary-bct4d" Oct 30 19:30:41.155391 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.155375 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g982p\" (UniqueName: \"kubernetes.io/projected/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kube-api-access-g982p\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.155434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.155395 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff89l\" (UniqueName: \"kubernetes.io/projected/651c14dc-17ea-4608-b927-ba1af3ea8b41-kube-api-access-ff89l\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.155434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.155402 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7xx\" (UniqueName: \"kubernetes.io/projected/eed84a66-cd93-4e58-adb3-e097e1b17a14-kube-api-access-nl7xx\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.156270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.156255 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6sq\" (UniqueName: \"kubernetes.io/projected/72fe1029-ecc8-4b99-9387-659ae6e414b3-kube-api-access-9b6sq\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.157632 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.157616 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:41.157673 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.157636 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:41.157673 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.157645 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.157735 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.157702 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.657690535 +0000 UTC m=+1.725526113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.158270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.158255 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqhrb\" (UniqueName: \"kubernetes.io/projected/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-kube-api-access-gqhrb\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.158344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.158322 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6vj\" (UniqueName: \"kubernetes.io/projected/a1c7a804-0740-4d51-884b-ea661daf80be-kube-api-access-9g6vj\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.158546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.158532 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6s9s\" (UniqueName: \"kubernetes.io/projected/b36aff52-af76-477f-990d-9900e4c43954-kube-api-access-r6s9s\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.160458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.160441 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65djh\" (UniqueName: \"kubernetes.io/projected/e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0-kube-api-access-65djh\") pod \"openshift-state-metrics-7bd7489957-6d6vf\" (UID: \"e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" Oct 30 19:30:41.160682 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.160666 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-575rj\" (UniqueName: \"kubernetes.io/projected/e8b76e60-de88-4c65-b570-fb4772dfbb5b-kube-api-access-575rj\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.243859 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243829 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.243859 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243862 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243881 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243895 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243913 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243949 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243962 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-log-socket\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243976 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243984 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-system-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243995 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-ovn\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.243970 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-tuning-conf-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244014 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-etc-kubernetes\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244003 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244017 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-sys-fs\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244052 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-k8s-cni-cncf-io\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244090 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244105 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-netns\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244122 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244148 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244172 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244191 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-conf-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244197 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-root\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244216 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244170 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-device-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244242 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244255 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-systemd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244269 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-conf\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244294 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244302 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-node-exporter-wtmp\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244312 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244328 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244351 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244370 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-systemd-units\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244374 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244391 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-kubelet-config\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-multus\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244406 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-node-log\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244399 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244440 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-run\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244457 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244493 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-lib-modules\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244517 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244528 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-system-cni-dir\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244533 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244550 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-multus-certs\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244572 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244578 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-var-lib-kubelet\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244594 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244611 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244635 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244636 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-netd\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244668 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-var-lib-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244675 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-kubelet\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244670 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-cni-dir\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244706 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244736 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244747 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-registration-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244762 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244785 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.244799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244811 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244831 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-cni-bin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244836 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244843 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysctl-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244838 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-cnibin\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244865 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-etc-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244869 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-run-openvswitch\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244878 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244908 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244930 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244951 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-systemd\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244954 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244972 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-modprobe-d\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244984 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/56834a9b-e103-4f55-b54f-14e73b4fa1db-dbus\") pod \"global-pull-secret-syncer-jllfr\" (UID: \"56834a9b-e103-4f55-b54f-14e73b4fa1db\") " pod="kube-system/global-pull-secret-syncer-jllfr" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.244984 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245024 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245030 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-multus-socket-dir-parent\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245045 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-sys\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245040 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245098 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245104 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce-host\") pod \"node-ca-7gh2g\" (UID: \"ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce\") " pod="openshift-image-registry/node-ca-7gh2g" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245100 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/651c14dc-17ea-4608-b927-ba1af3ea8b41-cnibin\") pod \"multus-additional-cni-plugins-l89fd\" (UID: \"651c14dc-17ea-4608-b927-ba1af3ea8b41\") " pod="openshift-multus/multus-additional-cni-plugins-l89fd" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245114 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245139 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-kubernetes\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245146 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245173 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245145 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245235 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245238 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245243 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-host\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245263 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245279 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245295 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245319 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245328 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245318 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-var-lib-kubelet\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245342 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245355 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-slash\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245357 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b36aff52-af76-477f-990d-9900e4c43954-host-cni-bin\") pod \"ovnkube-node-h6kbw\" (UID: \"b36aff52-af76-477f-990d-9900e4c43954\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245376 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245392 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e8b76e60-de88-4c65-b570-fb4772dfbb5b-etc-sysconfig\") pod \"tuned-2qlwk\" (UID: \"e8b76e60-de88-4c65-b570-fb4772dfbb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-2qlwk" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245368 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-hostroot\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245404 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245410 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1c7a804-0740-4d51-884b-ea661daf80be-sys\") pod \"node-exporter-tx4kh\" (UID: \"a1c7a804-0740-4d51-884b-ea661daf80be\") " pod="openshift-monitoring/node-exporter-tx4kh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245428 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245437 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-socket-dir\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245451 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245476 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245482 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eed84a66-cd93-4e58-adb3-e097e1b17a14-host-run-netns\") pod \"multus-2hcjh\" (UID: \"eed84a66-cd93-4e58-adb3-e097e1b17a14\") " pod="openshift-multus/multus-2hcjh" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245493 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245521 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245539 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5-etc-selinux\") pod \"aws-ebs-csi-driver-node-ng7cn\" (UID: \"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245542 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce67c2e9-3fdd-4a2b-b627-70993c44ea9f-hosts-file\") pod \"node-resolver-jcmd8\" (UID: \"ce67c2e9-3fdd-4a2b-b627-70993c44ea9f\") " pod="openshift-dns/node-resolver-jcmd8" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245573 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29bace5a-fb35-45e9-a2ce-41a61af06524-host-slash\") pod \"iptables-alerter-2pmgc\" (UID: \"29bace5a-fb35-45e9-a2ce-41a61af06524\") " pod="openshift-network-operator/iptables-alerter-2pmgc" Oct 30 19:30:41.246153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.245575 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/72fe1029-ecc8-4b99-9387-659ae6e414b3-crio-socket\") pod \"insights-runtime-extractor-nrp79\" (UID: \"72fe1029-ecc8-4b99-9387-659ae6e414b3\") " pod="openshift-insights/insights-runtime-extractor-nrp79" Oct 30 19:30:41.358567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.358535 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:41.649543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649507 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649561 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649613 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649654 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649696 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649673 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649704 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.649730 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649705 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649760 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649743719 +0000 UTC m=+2.717579265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649705 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649792 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649785385 +0000 UTC m=+2.717620930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649796 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649768 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:41.649884 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649735 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:41.650162 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649827 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649808506 +0000 UTC m=+2.717644055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:41.650162 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649921 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649904417 +0000 UTC m=+2.717739974 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:41.650162 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649935 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649929594 +0000 UTC m=+2.717765139 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:41.650162 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.649946 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.649940813 +0000 UTC m=+2.717776359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:41.751481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.751438 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:41.751647 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.751593 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:41.751647 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.751616 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:41.751647 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.751626 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.751762 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:41.751683 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.751670778 +0000 UTC m=+2.819506338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.772801 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.772747 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:41.915922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:41.915823 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:42.301099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.301007 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bjt85" Oct 30 19:30:42.406219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.406195 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kms8d" Oct 30 19:30:42.661868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.661819 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.661926 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.661962 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.661999 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662046 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.661960 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:42.662046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.662032 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662035 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662090 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662034 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662114 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662101 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.662077105 +0000 UTC m=+4.729912665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.662181 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662239 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.662221153 +0000 UTC m=+4.730056707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662261 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.662250705 +0000 UTC m=+4.730086254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662270 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662278 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.662268462 +0000 UTC m=+4.730104014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662300 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.66229048 +0000 UTC m=+4.730126029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:42.662340 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.662316 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.662307035 +0000 UTC m=+4.730142581 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:42.763482 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:42.763447 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:42.763627 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.763613 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:42.763663 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.763634 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:42.763663 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.763645 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:42.763749 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:42.763694 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.763681745 +0000 UTC m=+4.831517312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:43.692982 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:43.692951 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:30:43.998310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:43.998234 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:44.683322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683276 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.683322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683331 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683363 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683379 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683421 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683495 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683498 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683427 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683503 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.683484434 +0000 UTC m=+8.751319994 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683548 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683570 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.683557538 +0000 UTC m=+8.751393083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683506 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:44.683589 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683589 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.683578426 +0000 UTC m=+8.751413973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:44.684019 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683626 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.68360948 +0000 UTC m=+8.751445037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:44.684019 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683647 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.683640017 +0000 UTC m=+8.751475562 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:44.684019 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.683695 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.684019 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683767 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:44.684019 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.683805 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.683794606 +0000 UTC m=+8.751630169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:44.784775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.784740 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:44.785181 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.784913 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:44.785181 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.784936 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:44.785181 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.784948 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:44.785181 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:44.785008 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.784992522 +0000 UTC m=+8.852828073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:44.986180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:44.986094 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:46.094377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:46.094349 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:46.117427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:46.117394 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:47.190229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:47.190186 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:30:47.190737 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:47.190476 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:30:48.730847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.730806 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.730867 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.730890 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.730940 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.730962 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.730985 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731006 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731027 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731030 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731076 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731099 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731082164 +0000 UTC m=+16.798917713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731126 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731109623 +0000 UTC m=+16.798945174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.731147 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731196 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731189778 +0000 UTC m=+16.799025324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731208 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731202138 +0000 UTC m=+16.799037684 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731210 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731217 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731212427 +0000 UTC m=+16.799047973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:48.731354 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.731281 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.731267371 +0000 UTC m=+16.799102920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:48.832387 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:48.832344 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:48.832564 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.832509 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:48.832564 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.832529 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:48.832564 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.832539 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:48.832670 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:48.832596 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.832582326 +0000 UTC m=+16.900417872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:49.188843 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:49.188807 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:49.356672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:49.356636 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:51.292038 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:51.291944 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:30:51.382825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:51.382794 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:51.519790 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:51.519755 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.588502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:53.588469 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:53.732763 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:53.732726 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.034571 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.034532 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.177077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.177029 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:54.632238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.632201 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:54.633370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.633326 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="alertmanager" containerID="cri-o://f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7" gracePeriod=120 Oct 30 19:30:54.634298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.633841 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy" containerID="cri-o://5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac" gracePeriod=120 Oct 30 19:30:54.634298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.634020 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="prom-label-proxy" containerID="cri-o://4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7" gracePeriod=120 Oct 30 19:30:54.634298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.634131 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-metric" containerID="cri-o://4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa" gracePeriod=120 Oct 30 19:30:54.634298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.634202 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="config-reloader" containerID="cri-o://9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5" gracePeriod=120 Oct 30 19:30:54.634298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:54.634268 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-web" containerID="cri-o://26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909" gracePeriod=120 Oct 30 19:30:55.157592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157560 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7" exitCode=0 Oct 30 19:30:55.157592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157583 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa" exitCode=0 Oct 30 19:30:55.157592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157589 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac" exitCode=0 Oct 30 19:30:55.157592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157595 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5" exitCode=0 Oct 30 19:30:55.157592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157599 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7" exitCode=0 Oct 30 19:30:55.157877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157624 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7"} Oct 30 19:30:55.157877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157676 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa"} Oct 30 19:30:55.157877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157702 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac"} Oct 30 19:30:55.157877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157724 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5"} Oct 30 19:30:55.157877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.157748 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7"} Oct 30 19:30:55.772037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.772008 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:30:55.902070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.902031 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:55.998975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.998896 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.998975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.998956 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.998979 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999098 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999136 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999171 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999216 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999256 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999311 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999354 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999441 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999469 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999529 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") pod \"22cf1b54-38ba-446d-8083-b66ba618a7db\" (UID: \"22cf1b54-38ba-446d-8083-b66ba618a7db\") " Oct 30 19:30:55.999941 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:55.999890 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:56.000270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.000244 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:56.001367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.001290 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:56.002161 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.002116 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.002841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.002783 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "kube-api-access-4gtrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:56.003851 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.003821 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:56.004017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.003983 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.004259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.004231 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.004383 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.004323 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.004954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.004927 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:56.004954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.004946 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.007860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.007835 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.014025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.013970 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config" pod "22cf1b54-38ba-446d-8083-b66ba618a7db" (UID: "22cf1b54-38ba-446d-8083-b66ba618a7db"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:56.100473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100441 5733 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100470 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gtrr\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-kube-api-access-4gtrr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100485 5733 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-cluster-tls-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100499 5733 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100512 5733 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100524 5733 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100537 5733 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-web-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100551 5733 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-alertmanager-main-db\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100564 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22cf1b54-38ba-446d-8083-b66ba618a7db-metrics-client-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100576 5733 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/22cf1b54-38ba-446d-8083-b66ba618a7db-config-out\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100589 5733 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-main-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100603 5733 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/22cf1b54-38ba-446d-8083-b66ba618a7db-tls-assets\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.100671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.100615 5733 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/22cf1b54-38ba-446d-8083-b66ba618a7db-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:56.164969 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.164930 5733 generic.go:356] "Generic (PLEG): container finished" podID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerID="26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909" exitCode=0 Oct 30 19:30:56.165160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.165006 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909"} Oct 30 19:30:56.165160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.165036 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.165160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.165091 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"22cf1b54-38ba-446d-8083-b66ba618a7db","Type":"ContainerDied","Data":"bc0468473d014071abd973eefdfd3d1791fe13824ae112b26d61146fcb7e5ae4"} Oct 30 19:30:56.165160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.165127 5733 scope.go:122] "RemoveContainer" containerID="4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7" Oct 30 19:30:56.172808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.172792 5733 scope.go:122] "RemoveContainer" containerID="4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa" Oct 30 19:30:56.179751 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.179734 5733 scope.go:122] "RemoveContainer" containerID="5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac" Oct 30 19:30:56.186395 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.186380 5733 scope.go:122] "RemoveContainer" containerID="26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909" Oct 30 19:30:56.192884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.192869 5733 scope.go:122] "RemoveContainer" containerID="9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5" Oct 30 19:30:56.196154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.196131 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:56.201245 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.201217 5733 scope.go:122] "RemoveContainer" containerID="f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7" Oct 30 19:30:56.201370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.201353 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:56.207946 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.207924 5733 scope.go:122] "RemoveContainer" containerID="729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d" Oct 30 19:30:56.214550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.214534 5733 scope.go:122] "RemoveContainer" containerID="4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7" Oct 30 19:30:56.214813 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.214795 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7\": container with ID starting with 4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7 not found: ID does not exist" containerID="4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7" Oct 30 19:30:56.214853 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.214822 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7"} err="failed to get container status \"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7\": rpc error: code = NotFound desc = could not find container \"4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7\": container with ID starting with 4b65e1c17644c03e068a72194b6aef148b70279222b52bc633a87eff5deb47c7 not found: ID does not exist" Oct 30 19:30:56.214853 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.214851 5733 scope.go:122] "RemoveContainer" containerID="4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa" Oct 30 19:30:56.215081 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.215051 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa\": container with ID starting with 4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa not found: ID does not exist" containerID="4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa" Oct 30 19:30:56.215122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215089 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa"} err="failed to get container status \"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa\": rpc error: code = NotFound desc = could not find container \"4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa\": container with ID starting with 4b794211738a4d9d13663127a0349489dcbee4b72b6b9447303dba184dd48dfa not found: ID does not exist" Oct 30 19:30:56.215122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215105 5733 scope.go:122] "RemoveContainer" containerID="5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac" Oct 30 19:30:56.215338 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.215323 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac\": container with ID starting with 5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac not found: ID does not exist" containerID="5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac" Oct 30 19:30:56.215374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215342 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac"} err="failed to get container status \"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac\": rpc error: code = NotFound desc = could not find container \"5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac\": container with ID starting with 5d988be2c3557f109fe76bed9c1552f6e3404750559114b2cfdf35d3aa8655ac not found: ID does not exist" Oct 30 19:30:56.215374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215361 5733 scope.go:122] "RemoveContainer" containerID="26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909" Oct 30 19:30:56.215583 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.215567 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909\": container with ID starting with 26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909 not found: ID does not exist" containerID="26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909" Oct 30 19:30:56.215631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215597 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909"} err="failed to get container status \"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909\": rpc error: code = NotFound desc = could not find container \"26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909\": container with ID starting with 26695df2938577a31d55b8c8b6d815cadfe5828bdffa534b1b2f125afcc89909 not found: ID does not exist" Oct 30 19:30:56.215631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215613 5733 scope.go:122] "RemoveContainer" containerID="9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5" Oct 30 19:30:56.215835 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.215812 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5\": container with ID starting with 9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5 not found: ID does not exist" containerID="9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5" Oct 30 19:30:56.215891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215837 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5"} err="failed to get container status \"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5\": rpc error: code = NotFound desc = could not find container \"9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5\": container with ID starting with 9c30936d62cb799f27246b3f6356130c3230db419ac4886d58f7c2a438d07bd5 not found: ID does not exist" Oct 30 19:30:56.215891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.215848 5733 scope.go:122] "RemoveContainer" containerID="f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7" Oct 30 19:30:56.216150 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.216133 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7\": container with ID starting with f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7 not found: ID does not exist" containerID="f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7" Oct 30 19:30:56.216207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.216160 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7"} err="failed to get container status \"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7\": rpc error: code = NotFound desc = could not find container \"f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7\": container with ID starting with f47b3ef69113a6913afd2bdfdb6d4ea563eaff86124e734df085c504863044f7 not found: ID does not exist" Oct 30 19:30:56.216207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.216172 5733 scope.go:122] "RemoveContainer" containerID="729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d" Oct 30 19:30:56.216410 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.216395 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d\": container with ID starting with 729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d not found: ID does not exist" containerID="729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d" Oct 30 19:30:56.216448 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.216414 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d"} err="failed to get container status \"729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d\": rpc error: code = NotFound desc = could not find container \"729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d\": container with ID starting with 729c0abd781234f69f77ef89389059d0f6da7788533bc552a134fc586c53dd7d not found: ID does not exist" Oct 30 19:30:56.233530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233508 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:56.233816 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233800 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-web" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233819 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-web" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233835 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="init-config-reloader" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233843 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="init-config-reloader" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233865 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="alertmanager" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233873 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="alertmanager" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233892 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="prom-label-proxy" Oct 30 19:30:56.233905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233901 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="prom-label-proxy" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233921 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-metric" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233928 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-metric" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233944 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="config-reloader" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233951 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="config-reloader" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233964 5733 cpu_manager.go:414] "containerMap: removing container" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.233971 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234033 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="alertmanager" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234047 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="prom-label-proxy" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234072 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-metric" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234081 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="config-reloader" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234090 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234097 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="kube-rbac-proxy-web" Oct 30 19:30:56.234345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.234104 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" containerName="init-config-reloader" Oct 30 19:30:56.271180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.271103 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:56.271307 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.271182 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.276526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276488 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 19:30:56.276526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276504 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 19:30:56.276526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276522 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 19:30:56.276758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276488 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 19:30:56.276758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276620 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 19:30:56.276758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276629 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 19:30:56.276758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276708 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 19:30:56.276880 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.276789 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 19:30:56.277113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.277094 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wsp2p\"" Oct 30 19:30:56.280825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.280808 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 19:30:56.403144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403105 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-web-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403183 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403396 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403306 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403384 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403479 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403551 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403539 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403563 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403612 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403657 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403627 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403657 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403646 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403665 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqsm\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-kube-api-access-cbqsm\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403701 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-out\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.403800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.403788 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504655 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-out\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504702 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504721 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-web-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504764 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504784 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504803 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504897 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.504948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504925 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504951 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504981 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.504995 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.505011 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.505029 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqsm\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-kube-api-access-cbqsm\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.505125 5733 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:30:56.505344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.505275 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.505878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.505697 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.506262 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.506234 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.507949 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508010 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-out\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508183 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508190 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508265 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508435 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508355 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-config-volume\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.508617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.508601 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-web-config\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.509755 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.509738 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.518782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.518765 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqsm\" (UniqueName: \"kubernetes.io/projected/ad8b1c40-5211-4b2d-82a2-e11a686f5f60-kube-api-access-cbqsm\") pod \"alertmanager-main-0\" (UID: \"ad8b1c40-5211-4b2d-82a2-e11a686f5f60\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.673902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.673868 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 19:30:56.807312 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807279 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807323 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807397 5733 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807418 5733 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807463 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807473 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807455936 +0000 UTC m=+32.875291497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-serving-cert" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807495 5733 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807515 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807549 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807534479 +0000 UTC m=+32.875370041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807572 5733 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807579 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807568179 +0000 UTC m=+32.875403732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"service-ca" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807652 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807635625 +0000 UTC m=+32.875471180 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 19:30:56.807765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807747 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.808228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.807790 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.808228 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807854 5733 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 19:30:56.808228 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807879 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807871756 +0000 UTC m=+32.875707302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-config" not registered Oct 30 19:30:56.808228 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807888 5733 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:56.808228 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.807937 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.807924028 +0000 UTC m=+32.875759590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : object "openshift-console"/"console-oauth-config" not registered Oct 30 19:30:56.908259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.908224 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"console-6864546c55-lv6zs\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:30:56.908403 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.908387 5733 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 19:30:56.908441 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.908408 5733 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 19:30:56.908441 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.908417 5733 projected.go:196] Error preparing data for projected volume kube-api-access-p2sdb for pod openshift-console/console-6864546c55-lv6zs: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:56.908505 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:56.908468 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb podName:c6539c7b-9167-4d62-a086-6c02a73827d2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.908453303 +0000 UTC m=+32.976288855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p2sdb" (UniqueName: "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb") pod "console-6864546c55-lv6zs" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 19:30:56.997974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:56.997946 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 19:30:56.999270 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:30:56.999240 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8b1c40_5211_4b2d_82a2_e11a686f5f60.slice/crio-c2aa77fd0ae31d28340b308f42cc66866289cfdf7fccb96b4adfe42a92ad8b75 WatchSource:0}: Error finding container c2aa77fd0ae31d28340b308f42cc66866289cfdf7fccb96b4adfe42a92ad8b75: Status 404 returned error can't find the container with id c2aa77fd0ae31d28340b308f42cc66866289cfdf7fccb96b4adfe42a92ad8b75 Oct 30 19:30:57.169785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:57.169754 5733 generic.go:356] "Generic (PLEG): container finished" podID="ad8b1c40-5211-4b2d-82a2-e11a686f5f60" containerID="067010caf88b8b26002c69ba399d7b00371583599ea027fd99b569eba9437c25" exitCode=0 Oct 30 19:30:57.169955 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:57.169850 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerDied","Data":"067010caf88b8b26002c69ba399d7b00371583599ea027fd99b569eba9437c25"} Oct 30 19:30:57.169955 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:57.169909 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"c2aa77fd0ae31d28340b308f42cc66866289cfdf7fccb96b4adfe42a92ad8b75"} Oct 30 19:30:58.049135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.048220 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22cf1b54-38ba-446d-8083-b66ba618a7db" path="/var/lib/kubelet/pods/22cf1b54-38ba-446d-8083-b66ba618a7db/volumes" Oct 30 19:30:58.176976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.176938 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"bdfcdd25358f588d02b266ecefb3ef7438cc6687a08f5a180f0313bfb06319eb"} Oct 30 19:30:58.177149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.177019 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"bd37a2075d8116b45d57e26de8f8fa1d47a8a0f61dbc491a52bdfd7b02dd9070"} Oct 30 19:30:58.177149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.177075 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"803cbd7611763df57ad6df5db5ebaf0c37e2f8f85e89fd810729222906c0ffc3"} Oct 30 19:30:58.177149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.177113 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"7413d76bf617d4676f945d4cdb2be19d8198517b817a51c54d96aa4af37691a0"} Oct 30 19:30:58.177289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.177154 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"bbc9cc0041922147d675ce6058b3b4498522cc9984fbe151b52402af7f1dde9b"} Oct 30 19:30:58.177289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.177188 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ad8b1c40-5211-4b2d-82a2-e11a686f5f60","Type":"ContainerStarted","Data":"9b680c923e6f1f426e5394088614d7f7e84284b40199383f84d784b4c86415fb"} Oct 30 19:30:58.212907 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:58.212852 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.192525743 podStartE2EDuration="2.212838343s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.3126ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:58.21192167 +0000 UTC m=+18.279757239" watchObservedRunningTime="2026-10-30 19:30:58.212838343 +0000 UTC m=+18.280673911" Oct 30 19:30:59.075230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075190 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:30:59.075614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075497 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="prometheus" containerID="cri-o://1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" gracePeriod=600 Oct 30 19:30:59.075614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075525 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="thanos-sidecar" containerID="cri-o://d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" gracePeriod=600 Oct 30 19:30:59.075614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075550 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" gracePeriod=600 Oct 30 19:30:59.075614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075573 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="config-reloader" containerID="cri-o://e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" gracePeriod=600 Oct 30 19:30:59.075834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075518 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy" containerID="cri-o://9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" gracePeriod=600 Oct 30 19:30:59.075834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.075550 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-web" containerID="cri-o://e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" gracePeriod=600 Oct 30 19:30:59.361798 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.361773 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:30:59.376128 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:30:59.376097 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:30:59.440073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440027 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440077 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440096 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440120 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440140 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440160 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440211 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440247 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440272 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440314 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440349 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440399 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440444 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440477 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440511 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440552 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440582 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440639 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.440704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440680 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.441221 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440711 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") pod \"d6376030-0944-400b-942f-286dc1cce30b\" (UID: \"d6376030-0944-400b-942f-286dc1cce30b\") " Oct 30 19:30:59.443148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440545 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.440582 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.441516 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.441946 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.442844 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.442943 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.443274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.443116 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.443337 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.443483 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:59.443603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.443566 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.443827 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.443799 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.444494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.444193 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.444794 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.444755 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:30:59.445666 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.445638 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "kube-api-access-pxbkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:30:59.445768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.445746 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.445884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.445860 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.446211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.446186 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.446298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.446232 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:30:59.447198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.447177 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.454458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.454437 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config" pod "d6376030-0944-400b-942f-286dc1cce30b" (UID: "d6376030-0944-400b-942f-286dc1cce30b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:30:59.542400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542357 5733 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-kube-rbac-proxy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542392 5733 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-web-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542406 5733 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-grpc-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542422 5733 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542436 5733 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-db\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542452 5733 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542464 5733 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542477 5733 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542490 5733 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542503 5733 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542517 5733 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-metrics-client-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542530 5733 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542543 5733 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d6376030-0944-400b-942f-286dc1cce30b-config-out\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542556 5733 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542572 5733 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-configmap-metrics-client-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542590 5733 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d6376030-0944-400b-942f-286dc1cce30b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542603 5733 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542617 5733 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-tls-assets\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542631 5733 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6376030-0944-400b-942f-286dc1cce30b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:30:59.542693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:30:59.542643 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxbkf\" (UniqueName: \"kubernetes.io/projected/d6376030-0944-400b-942f-286dc1cce30b-kube-api-access-pxbkf\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:00.186950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186912 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" exitCode=0 Oct 30 19:31:00.186950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186945 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" exitCode=0 Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186957 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" exitCode=0 Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186967 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" exitCode=0 Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186977 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" exitCode=0 Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186986 5733 generic.go:356] "Generic (PLEG): container finished" podID="d6376030-0944-400b-942f-286dc1cce30b" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" exitCode=0 Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.186987 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187028 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187095 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187125 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187152 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187165 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187177 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187202 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} Oct 30 19:31:00.187381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.187226 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d6376030-0944-400b-942f-286dc1cce30b","Type":"ContainerDied","Data":"d52206c1af0e59e0d92c153834429de61cbb308ec221adaf424d76f018c45ca3"} Oct 30 19:31:00.194887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.194869 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.201995 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.201977 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.208639 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.208619 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.214191 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.214144 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:31:00.217222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.217185 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.222914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.222889 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:31:00.225170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.225155 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.232644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.232615 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.239424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.239406 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.239709 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.239689 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.239782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.239717 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.239782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.239736 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.239983 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.239968 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.240023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.239988 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.240023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240002 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.240225 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.240204 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.240304 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240233 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.240304 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240255 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.240512 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.240493 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.240590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240513 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.240590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240525 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.240737 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.240718 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.240774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240741 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.240774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240753 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.240945 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.240929 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.240983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240950 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.240983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.240968 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.241271 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:00.241255 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.241342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241272 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.241342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241285 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.241522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241503 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.241577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241523 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.241774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241757 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.241774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241773 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.241999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241977 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.242037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.241999 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.242232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242217 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.242282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242232 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.242413 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242398 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.242453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242413 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.242588 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242573 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.242625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242587 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.242799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242781 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.242799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.242798 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.243016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243001 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.243016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243015 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.243256 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243234 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.243256 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243252 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.243476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243459 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.243476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243474 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.243729 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243706 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.243729 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243728 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.243956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243937 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.243956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.243954 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.244170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244152 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.244170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244169 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.244377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244355 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.244439 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244377 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.244590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244573 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.244590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244589 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.244837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244817 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.244837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.244836 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.245128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245103 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.245203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245128 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.245399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245369 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.245399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245398 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.245632 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245614 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.245632 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245631 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.245841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245819 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.245906 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.245845 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.246043 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246023 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.246043 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246043 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.246259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246241 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.246259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246257 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.246457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246439 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.246457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246456 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.246636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246617 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.246697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246637 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.246835 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246814 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.246835 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.246832 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.247151 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247025 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.247151 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247076 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.247380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247355 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.247428 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247383 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.247773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247751 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.247773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.247772 5733 scope.go:122] "RemoveContainer" containerID="334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5" Oct 30 19:31:00.248076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248040 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5"} err="failed to get container status \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": rpc error: code = NotFound desc = could not find container \"334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5\": container with ID starting with 334177697fd305fb562ff9c6bdbf5dbe485ef864fe866fa7c3497bb1a24c6cd5 not found: ID does not exist" Oct 30 19:31:00.248076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248074 5733 scope.go:122] "RemoveContainer" containerID="9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7" Oct 30 19:31:00.248320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248298 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7"} err="failed to get container status \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": rpc error: code = NotFound desc = could not find container \"9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7\": container with ID starting with 9ed5a997942f03565734cab5049de12544aacfe2b9c43bfd404408256c7790f7 not found: ID does not exist" Oct 30 19:31:00.248390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248321 5733 scope.go:122] "RemoveContainer" containerID="e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1" Oct 30 19:31:00.248538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248523 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1"} err="failed to get container status \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": rpc error: code = NotFound desc = could not find container \"e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1\": container with ID starting with e2d8002327cf3eb9fd7ade1864ff22cd53cfe07f2d49a5e4e5e4fc76f440b2f1 not found: ID does not exist" Oct 30 19:31:00.248583 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248537 5733 scope.go:122] "RemoveContainer" containerID="d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd" Oct 30 19:31:00.248793 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248769 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd"} err="failed to get container status \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": rpc error: code = NotFound desc = could not find container \"d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd\": container with ID starting with d89f5a746ab6493aa7d4f60b9b380b46cb9304782184630ee2c85f74723d06fd not found: ID does not exist" Oct 30 19:31:00.248832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.248795 5733 scope.go:122] "RemoveContainer" containerID="e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422" Oct 30 19:31:00.249042 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.249026 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422"} err="failed to get container status \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": rpc error: code = NotFound desc = could not find container \"e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422\": container with ID starting with e242e47f53130717544d1de8c492701577b0343ba20cf1a93e08c06db950c422 not found: ID does not exist" Oct 30 19:31:00.249113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.249043 5733 scope.go:122] "RemoveContainer" containerID="1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0" Oct 30 19:31:00.249261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.249244 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0"} err="failed to get container status \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": rpc error: code = NotFound desc = could not find container \"1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0\": container with ID starting with 1a81405a2a1456559ff555e677cd90b0751e288eb45b9e07db7f4815e8f358b0 not found: ID does not exist" Oct 30 19:31:00.249303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.249260 5733 scope.go:122] "RemoveContainer" containerID="ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10" Oct 30 19:31:00.249457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.249441 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10"} err="failed to get container status \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": rpc error: code = NotFound desc = could not find container \"ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10\": container with ID starting with ac925422d79d8f13712e267e3a45be20d054d761481d87827f6f63661a019c10 not found: ID does not exist" Oct 30 19:31:00.258664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258646 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:31:00.258894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258882 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258894 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258905 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-thanos" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258910 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-thanos" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258917 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="thanos-sidecar" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258922 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="thanos-sidecar" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258941 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="prometheus" Oct 30 19:31:00.258948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258946 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="prometheus" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258953 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-web" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258958 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-web" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258965 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258969 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258980 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="init-config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.258985 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="init-config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259021 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="prometheus" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259030 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-web" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259036 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259043 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="init-config-reloader" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259049 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259069 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="kube-rbac-proxy-thanos" Oct 30 19:31:00.259216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.259076 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6376030-0944-400b-942f-286dc1cce30b" containerName="thanos-sidecar" Oct 30 19:31:00.287701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.287674 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:31:00.287860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.287839 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.294011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.293973 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 19:31:00.294272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294179 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 19:31:00.294272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294204 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 19:31:00.294272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294212 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 19:31:00.294272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294205 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294363 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294406 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294417 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294446 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294466 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 19:31:00.294663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.294513 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 19:31:00.295862 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.295847 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5kn8\"" Oct 30 19:31:00.296379 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.296364 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 19:31:00.296680 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.296663 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6b35nveaduvtq\"" Oct 30 19:31:00.303501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.303473 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 19:31:00.303831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.303817 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 19:31:00.347609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347576 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347620 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347644 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347745 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347774 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347798 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347843 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347870 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347959 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347939 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347988 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347963 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.347988 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.347983 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348048 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348013 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348115 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348102 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348125 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348140 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wszn\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-kube-api-access-7wszn\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348240 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348168 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348318 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348378 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348448 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348436 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.348484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.348454 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.448996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.448958 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.448996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.448994 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449015 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449077 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449104 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449131 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449155 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449231 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449262 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449283 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wszn\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-kube-api-access-7wszn\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449324 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449392 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449436 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449465 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449491 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449517 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449545 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449569 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449628 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.449664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449659 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.450337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.449980 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.450337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.450198 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.450661 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.450634 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.450752 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.450681 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.451297 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.451269 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.451468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.451448 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.452783 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.452362 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.452783 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.452504 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.453729 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.453581 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.454148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.453955 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.454148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.454108 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.454290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.454187 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ef38f10e-731b-49df-a5c8-0fce38cc59ab-config-out\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.454520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.454396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ef38f10e-731b-49df-a5c8-0fce38cc59ab-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.458976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.454771 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.458976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.454904 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.458976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.455408 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.458976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.455895 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.458976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.456540 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-web-config\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.459881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.459856 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ef38f10e-731b-49df-a5c8-0fce38cc59ab-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.466440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.466418 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wszn\" (UniqueName: \"kubernetes.io/projected/ef38f10e-731b-49df-a5c8-0fce38cc59ab-kube-api-access-7wszn\") pod \"prometheus-k8s-0\" (UID: \"ef38f10e-731b-49df-a5c8-0fce38cc59ab\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:00.693314 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:00.693218 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:01.016473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.016435 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 19:31:01.020410 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:31:01.020386 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef38f10e_731b_49df_a5c8_0fce38cc59ab.slice/crio-2ccf8e23e542f481cc3148be5fe1f02048c45e4473eaadc3b18bb20c717fd1c7 WatchSource:0}: Error finding container 2ccf8e23e542f481cc3148be5fe1f02048c45e4473eaadc3b18bb20c717fd1c7: Status 404 returned error can't find the container with id 2ccf8e23e542f481cc3148be5fe1f02048c45e4473eaadc3b18bb20c717fd1c7 Oct 30 19:31:01.192479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.192448 5733 generic.go:356] "Generic (PLEG): container finished" podID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerID="f11d9a2c6c3d72019edda90d802edaadb488b50c5eb50a5daf6216ae08d35be5" exitCode=0 Oct 30 19:31:01.192851 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.192489 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerDied","Data":"f11d9a2c6c3d72019edda90d802edaadb488b50c5eb50a5daf6216ae08d35be5"} Oct 30 19:31:01.192851 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.192536 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"2ccf8e23e542f481cc3148be5fe1f02048c45e4473eaadc3b18bb20c717fd1c7"} Oct 30 19:31:01.363626 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.363596 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:31:01.367551 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:01.367529 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 19:31:02.047214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.047134 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6376030-0944-400b-942f-286dc1cce30b" path="/var/lib/kubelet/pods/d6376030-0944-400b-942f-286dc1cce30b/volumes" Oct 30 19:31:02.198745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198708 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"2e8b1d279486eff1aa5ad0158d8cbf6aa793dccff022b9e5db052fa3b368ba74"} Oct 30 19:31:02.199140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198776 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"0b054e6dfbba7f847e0ff3d3f0f45334891b05096af2dc64fcf2fa335e790db7"} Oct 30 19:31:02.199140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198803 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"aa8c71736498af427b2124d495895c7b10861a0461bdd895e8cf78efa98f1327"} Oct 30 19:31:02.199140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198828 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"30e5a9ded9ef3f960ca8bcb8c13b7e64a8e9076aa281344fe4aefd7664b91372"} Oct 30 19:31:02.199140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198858 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"4eba4e6c9723a4c516785d190ea13cd7a5ec6c792e61949d01f68bacd1adcf48"} Oct 30 19:31:02.199140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.198880 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ef38f10e-731b-49df-a5c8-0fce38cc59ab","Type":"ContainerStarted","Data":"ab0c3f4950bbab136105f0bb4f29773a971937ba1e95fc4fd29919a61d734a1b"} Oct 30 19:31:02.237331 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.237267 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.2090739839999998 podStartE2EDuration="2.237248804s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.17482ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:02.23617066 +0000 UTC m=+22.304006229" watchObservedRunningTime="2026-10-30 19:31:02.237248804 +0000 UTC m=+22.305084374" Oct 30 19:31:02.900558 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:02.900515 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:03.038396 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:03.038363 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.673699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:04.673661 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:04.818397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:04.818362 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:05.693628 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:05.693596 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:06.045300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.045217 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6864546c55-lv6zs" podUID="c6539c7b-9167-4d62-a086-6c02a73827d2" containerName="console" containerID="cri-o://047baa3ddc2d1809f035e106139e5a0b8132291dbfe5fd321e445245535dc3e0" gracePeriod=15 Oct 30 19:31:06.213855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.213825 5733 generic.go:356] "Generic (PLEG): container finished" podID="c6539c7b-9167-4d62-a086-6c02a73827d2" containerID="047baa3ddc2d1809f035e106139e5a0b8132291dbfe5fd321e445245535dc3e0" exitCode=0 Oct 30 19:31:06.214010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.213904 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6864546c55-lv6zs" event={"ID":"c6539c7b-9167-4d62-a086-6c02a73827d2","Type":"ContainerDied","Data":"047baa3ddc2d1809f035e106139e5a0b8132291dbfe5fd321e445245535dc3e0"} Oct 30 19:31:06.312178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.312115 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:31:06.409942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.409910 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.409942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.409943 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.410203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.409970 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410506 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410562 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410779 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410794 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410829 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410878 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410923 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") pod \"c6539c7b-9167-4d62-a086-6c02a73827d2\" (UID: \"c6539c7b-9167-4d62-a086-6c02a73827d2\") " Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.411365 5733 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-oauth-serving-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.411387 5733 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6539c7b-9167-4d62-a086-6c02a73827d2-tmp\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.410822 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.411711 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:06.412718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.412096 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:31:06.413985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.413199 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "kube-api-access-p2sdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:31:06.415678 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.415640 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:06.415825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.415745 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config" pod "c6539c7b-9167-4d62-a086-6c02a73827d2" (UID: "c6539c7b-9167-4d62-a086-6c02a73827d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:31:06.512485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512446 5733 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-trusted-ca-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.512485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512481 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2sdb\" (UniqueName: \"kubernetes.io/projected/c6539c7b-9167-4d62-a086-6c02a73827d2-kube-api-access-p2sdb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.512485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512492 5733 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-oauth-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.512719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512501 5733 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6539c7b-9167-4d62-a086-6c02a73827d2-console-serving-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.512719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512511 5733 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:06.512719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:06.512520 5733 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6539c7b-9167-4d62-a086-6c02a73827d2-console-config\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:31:07.218669 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:07.218637 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6864546c55-lv6zs" event={"ID":"c6539c7b-9167-4d62-a086-6c02a73827d2","Type":"ContainerDied","Data":"78075e7633b90b03b0f161469aee0ef8492042389b81f33cba173dcef4bd9e31"} Oct 30 19:31:07.219045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:07.218675 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6864546c55-lv6zs" Oct 30 19:31:07.219045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:07.218701 5733 scope.go:122] "RemoveContainer" containerID="047baa3ddc2d1809f035e106139e5a0b8132291dbfe5fd321e445245535dc3e0" Oct 30 19:31:07.241848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:07.241825 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:31:07.250839 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:07.250811 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6864546c55-lv6zs"] Oct 30 19:31:08.046717 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:08.046685 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6539c7b-9167-4d62-a086-6c02a73827d2" path="/var/lib/kubelet/pods/c6539c7b-9167-4d62-a086-6c02a73827d2/volumes" Oct 30 19:31:08.447841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:08.447817 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" Oct 30 19:31:11.295238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:11.295208 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrpl8" Oct 30 19:31:13.189402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:13.189367 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:13.204676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:13.204648 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:13.257323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:13.257300 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 19:31:16.360263 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:16.360208 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bct4d_0678842b-b564-4b68-ae5b-aaffd1680479/serve-healthcheck-canary/0.log" Oct 30 19:31:16.602452 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:16.602417 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.784484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:16.784383 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.880579 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:18.880549 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.048653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:19.048620 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.272771 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:19.272693 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.409068 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:19.409024 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.116612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:27.116571 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.263228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:27.263191 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.698928 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:27.698893 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.834536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:27.834504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.374500 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:29.374468 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:31:29.408111 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:29.408084 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.545595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:29.545565 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:36.388021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:36.387982 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:36.538846 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:36.538806 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:49.240299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:31:49.240267 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.375461 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:31:59.375419 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:32:29.375615 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:32:29.375525 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:32:43.555971 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:32:43.555930 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:43.696052 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:32:43.696020 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:49.304089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:32:49.304033 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:59.375570 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:32:59.375530 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:32:59.550419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:32:59.550386 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.386364 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:15.386328 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.533719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:15.533690 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:29.375521 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:33:29.375485 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:33:42.691878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:42.691810 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:42.828840 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:42.828808 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:47.606352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:47.606314 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:47.786546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:47.786516 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:47.938877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:47.938789 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.361824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:49.361792 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.585434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:49.585397 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:49.735590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:49.735513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:50.916076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.915991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:50.973781 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.973748 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww"] Oct 30 19:33:50.974069 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.974041 5733 cpu_manager.go:414] "containerMap: removing container" podUID="c6539c7b-9167-4d62-a086-6c02a73827d2" containerName="console" Oct 30 19:33:50.974069 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.974067 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6539c7b-9167-4d62-a086-6c02a73827d2" containerName="console" Oct 30 19:33:50.974149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.974113 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6539c7b-9167-4d62-a086-6c02a73827d2" containerName="console" Oct 30 19:33:50.976184 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.976168 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:50.982569 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.982553 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:33:50.982644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.982632 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-qvslp\"" Oct 30 19:33:50.982761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.982745 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:33:50.989322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:50.989302 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww"] Oct 30 19:33:51.032049 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.032022 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kff5v\" (UniqueName: \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.032178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.032079 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.032226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.032181 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.067529 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.067505 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5"] Oct 30 19:33:51.069657 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.069642 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.074977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.074957 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-7fc2m\"" Oct 30 19:33:51.079874 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.079851 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5"] Oct 30 19:33:51.132815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.132789 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.132923 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.132827 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.132923 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.132910 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hth9n\" (UniqueName: \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.133017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.132944 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.133017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.132980 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kff5v\" (UniqueName: \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.133151 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.133098 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.133204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.133177 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.133401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.133381 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.146346 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.146328 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kff5v\" (UniqueName: \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.171211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.171161 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f"] Oct 30 19:33:51.173300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.173286 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.178762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.178746 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-c9d46\"" Oct 30 19:33:51.184860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.184837 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f"] Oct 30 19:33:51.234301 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234273 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.234430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234322 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.234430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234341 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.234430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234361 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z2zs\" (UniqueName: \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.234597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234550 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hth9n\" (UniqueName: \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.234658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234606 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.234701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234667 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.234870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.234855 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.247947 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.247921 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hth9n\" (UniqueName: \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.272814 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.272793 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf"] Oct 30 19:33:51.275046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.275033 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.279715 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.279693 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ksjcr\"" Oct 30 19:33:51.285213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.285193 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf"] Oct 30 19:33:51.288245 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.288226 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335481 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp58p\" (UniqueName: \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335542 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335592 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335621 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335647 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.335689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.335680 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z2zs\" (UniqueName: \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.336358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.336190 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.336468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.336441 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.352727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.352669 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z2zs\" (UniqueName: \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.381522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.381498 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:33:51.436496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.436460 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.436630 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.436547 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.436693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.436648 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp58p\" (UniqueName: \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.436912 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.436888 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.437145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.437126 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.450535 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.450509 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp58p\" (UniqueName: \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.490467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.490445 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:33:51.562390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.562360 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww"] Oct 30 19:33:51.565302 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:51.565261 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb4f8fd_3386_4d32_ae7e_397431ef010d.slice/crio-bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303 WatchSource:0}: Error finding container bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303: Status 404 returned error can't find the container with id bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303 Oct 30 19:33:51.567252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.567236 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:33:51.588427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.588407 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:33:51.654049 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.654018 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5"] Oct 30 19:33:51.657193 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:51.657160 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee56d4bd_99c3_4e04_8e7a_ca5011951c65.slice/crio-37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819 WatchSource:0}: Error finding container 37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819: Status 404 returned error can't find the container with id 37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819 Oct 30 19:33:51.758845 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.758778 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f"] Oct 30 19:33:51.761663 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:51.761634 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c1363e7_7ee6_4bcc_865e_169343b35123.slice/crio-c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504 WatchSource:0}: Error finding container c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504: Status 404 returned error can't find the container with id c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504 Oct 30 19:33:51.766870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.766850 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" event={"ID":"ee56d4bd-99c3-4e04-8e7a-ca5011951c65","Type":"ContainerStarted","Data":"37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819"} Oct 30 19:33:51.767962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.767943 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" event={"ID":"7eb4f8fd-3386-4d32-ae7e-397431ef010d","Type":"ContainerStarted","Data":"bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303"} Oct 30 19:33:51.769004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.768986 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" event={"ID":"4c1363e7-7ee6-4bcc-865e-169343b35123","Type":"ContainerStarted","Data":"c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504"} Oct 30 19:33:51.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:51.860152 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf"] Oct 30 19:33:51.863265 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:51.863241 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab300d5_80cb_47b0_aacc_0cf91fb86e5f.slice/crio-56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba WatchSource:0}: Error finding container 56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba: Status 404 returned error can't find the container with id 56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba Oct 30 19:33:52.228880 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.228847 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb"] Oct 30 19:33:52.231662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.231643 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.236972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.236951 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-vn89d\"" Oct 30 19:33:52.241994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.241973 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb"] Oct 30 19:33:52.244365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.244154 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.244365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.244201 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t4lx\" (UniqueName: \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.244365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.244239 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.344809 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.344769 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.344997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.344828 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t4lx\" (UniqueName: \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.344997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.344856 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.345329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.345304 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.345599 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.345573 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.360093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.360050 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t4lx\" (UniqueName: \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.549776 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.549678 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:33:52.776635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.776598 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" event={"ID":"bab300d5-80cb-47b0-aacc-0cf91fb86e5f","Type":"ContainerStarted","Data":"56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba"} Oct 30 19:33:52.844089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:52.844047 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb"] Oct 30 19:33:52.857536 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:52.857506 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4559fb0d_c4cc_444f_998c_176d280d8efa.slice/crio-acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90 WatchSource:0}: Error finding container acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90: Status 404 returned error can't find the container with id acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90 Oct 30 19:33:53.399387 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.399348 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft"] Oct 30 19:33:53.402436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.402414 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.409429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.409392 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-v2hvt\"" Oct 30 19:33:53.413867 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.413775 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft"] Oct 30 19:33:53.454199 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.454165 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74hq\" (UniqueName: \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.454375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.454248 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.454375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.454277 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.555633 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.555589 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74hq\" (UniqueName: \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.555817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.555646 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.555817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.555674 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.556016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.555996 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.556578 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.556556 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.579489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.579462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74hq\" (UniqueName: \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.707951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.707876 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:53.721258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.721234 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:33:53.786008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.785972 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerStarted","Data":"acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90"} Oct 30 19:33:53.886340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:53.886300 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:54.054105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:54.053932 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft"] Oct 30 19:33:54.062077 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:54.062021 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0980cf99_c017_4662_bca0_c4ea2be970d4.slice/crio-7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3 WatchSource:0}: Error finding container 7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3: Status 404 returned error can't find the container with id 7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3 Oct 30 19:33:54.794564 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:54.794528 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" event={"ID":"0980cf99-c017-4662-bca0-c4ea2be970d4","Type":"ContainerStarted","Data":"7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3"} Oct 30 19:33:55.110081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:55.109407 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:55.254610 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:55.254527 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:55.391597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:55.391420 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:56.381744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.380308 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs"] Oct 30 19:33:56.389612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.389244 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.397744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.397141 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs"] Oct 30 19:33:56.397744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.397486 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9db9c\"" Oct 30 19:33:56.485372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.485081 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.485372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.485165 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.485372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.485209 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrvr\" (UniqueName: \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.586822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.585630 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.586822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.585710 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.586822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.585750 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrvr\" (UniqueName: \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.586822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.586514 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.586822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.586771 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.628976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.628943 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrvr\" (UniqueName: \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:56.712348 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:56.712027 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:33:57.057589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:57.057543 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs"] Oct 30 19:33:57.068858 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:57.068826 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04482908_bc6a_4e88_9aa3_1090f24e7ebf.slice/crio-0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62 WatchSource:0}: Error finding container 0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62: Status 404 returned error can't find the container with id 0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62 Oct 30 19:33:57.821284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:57.821195 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" event={"ID":"04482908-bc6a-4e88-9aa3-1090f24e7ebf","Type":"ContainerStarted","Data":"0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62"} Oct 30 19:33:57.937904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:57.937785 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:59.131429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.129433 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r"] Oct 30 19:33:59.135476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.134910 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.141780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.141735 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r"] Oct 30 19:33:59.165483 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.165455 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657"] Oct 30 19:33:59.169538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.169514 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.180566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.180526 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657"] Oct 30 19:33:59.196894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.196864 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9"] Oct 30 19:33:59.201415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.200886 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.214593 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.214555 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9"] Oct 30 19:33:59.215258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215034 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qh4j\" (UniqueName: \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.215258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215117 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.215258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215200 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84t88\" (UniqueName: \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.215467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215260 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.215467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215339 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.215467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.215369 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316121 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316205 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316265 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316314 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qh4j\" (UniqueName: \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316349 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316404 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwghq\" (UniqueName: \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316457 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316490 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84t88\" (UniqueName: \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.316534 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.317001 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.317462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.318486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.318130 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.319397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.319371 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.346899 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.346869 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qh4j\" (UniqueName: \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.348881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.348851 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84t88\" (UniqueName: \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.395304 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:33:59.395170 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:33:59.418605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.418002 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwghq\" (UniqueName: \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.418605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.418362 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.418605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.418439 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.421251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.420512 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.424008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.423427 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.437679 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.437660 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwghq\" (UniqueName: \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.456078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.456020 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:33:59.500440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.500147 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:33:59.535557 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.535281 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:33:59.776650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.776611 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r"] Oct 30 19:33:59.784315 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:59.784272 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5931d0ca_c1d3_4950_a8b7_b176d2b7e351.slice/crio-84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d WatchSource:0}: Error finding container 84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d: Status 404 returned error can't find the container with id 84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d Oct 30 19:33:59.811674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.811646 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657"] Oct 30 19:33:59.840630 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.840565 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" event={"ID":"8403f4ea-8e42-41b3-80a1-517d0a8af80b","Type":"ContainerStarted","Data":"c7e75b3c4b3dc9004f0f64b8775700fb69f2654174622b50d3393484f4ba9771"} Oct 30 19:33:59.842643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.842576 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" event={"ID":"5931d0ca-c1d3-4950-a8b7-b176d2b7e351","Type":"ContainerStarted","Data":"84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d"} Oct 30 19:33:59.875824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:33:59.875791 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9"] Oct 30 19:33:59.881918 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:33:59.881889 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07707c3_3d89_42b4_a46c_a569fd234f7b.slice/crio-24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8 WatchSource:0}: Error finding container 24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8: Status 404 returned error can't find the container with id 24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8 Oct 30 19:34:00.852794 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:00.852689 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" event={"ID":"f07707c3-3d89-42b4-a46c-a569fd234f7b","Type":"ContainerStarted","Data":"24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8"} Oct 30 19:34:07.932488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.932445 5733 generic.go:356] "Generic (PLEG): container finished" podID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerID="68217742401ec23772767c66e0f1cf8a1097f39d18a17dbfaf0842408013200a" exitCode=0 Oct 30 19:34:07.932977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.932545 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" event={"ID":"ee56d4bd-99c3-4e04-8e7a-ca5011951c65","Type":"ContainerDied","Data":"68217742401ec23772767c66e0f1cf8a1097f39d18a17dbfaf0842408013200a"} Oct 30 19:34:07.935996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.935960 5733 generic.go:356] "Generic (PLEG): container finished" podID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerID="a4a3194324ca19aa63b52e54d09d3f2c499df441e6ccce7478ca6e8a07572d9e" exitCode=0 Oct 30 19:34:07.936106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.936080 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" event={"ID":"7eb4f8fd-3386-4d32-ae7e-397431ef010d","Type":"ContainerDied","Data":"a4a3194324ca19aa63b52e54d09d3f2c499df441e6ccce7478ca6e8a07572d9e"} Oct 30 19:34:07.941180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.941147 5733 generic.go:356] "Generic (PLEG): container finished" podID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerID="2d74c1d197d406a898b909542f9d5b4554c83fc5ed1215f39df69de71f21c248" exitCode=0 Oct 30 19:34:07.941401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.941377 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" event={"ID":"bab300d5-80cb-47b0-aacc-0cf91fb86e5f","Type":"ContainerDied","Data":"2d74c1d197d406a898b909542f9d5b4554c83fc5ed1215f39df69de71f21c248"} Oct 30 19:34:07.945403 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.945308 5733 generic.go:356] "Generic (PLEG): container finished" podID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerID="f8e51bd148088cd5398aa31ab0fba786181fabcd4f6ef7ecb320e12d7ee8a57d" exitCode=0 Oct 30 19:34:07.945403 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:07.945361 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" event={"ID":"04482908-bc6a-4e88-9aa3-1090f24e7ebf","Type":"ContainerDied","Data":"f8e51bd148088cd5398aa31ab0fba786181fabcd4f6ef7ecb320e12d7ee8a57d"} Oct 30 19:34:08.955223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:08.954152 5733 generic.go:356] "Generic (PLEG): container finished" podID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerID="39da2cf58dd9e5bfdb102844a263f34d830b22eb367e4c5cc988c494305ba03e" exitCode=0 Oct 30 19:34:08.955223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:08.954323 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerDied","Data":"39da2cf58dd9e5bfdb102844a263f34d830b22eb367e4c5cc988c494305ba03e"} Oct 30 19:34:08.960828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:08.959339 5733 generic.go:356] "Generic (PLEG): container finished" podID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerID="48bc2d3a1fddf43ed9b023b606ac1c7925cad4760a9fde27f6fbc9f5b64eb7dc" exitCode=0 Oct 30 19:34:08.960828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:08.959374 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" event={"ID":"4c1363e7-7ee6-4bcc-865e-169343b35123","Type":"ContainerDied","Data":"48bc2d3a1fddf43ed9b023b606ac1c7925cad4760a9fde27f6fbc9f5b64eb7dc"} Oct 30 19:34:09.966852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.966818 5733 generic.go:356] "Generic (PLEG): container finished" podID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerID="dada43f8706867fe5d4655b0e20685bbf3170b06b1559a3f862a6d70f4104844" exitCode=0 Oct 30 19:34:09.967345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.966905 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" event={"ID":"ee56d4bd-99c3-4e04-8e7a-ca5011951c65","Type":"ContainerDied","Data":"dada43f8706867fe5d4655b0e20685bbf3170b06b1559a3f862a6d70f4104844"} Oct 30 19:34:09.969543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.969520 5733 generic.go:356] "Generic (PLEG): container finished" podID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerID="8b3c41a0ba870922f464b0136df69815e7c529ed33852083c132b9a8cd4c239b" exitCode=0 Oct 30 19:34:09.969639 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.969621 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" event={"ID":"7eb4f8fd-3386-4d32-ae7e-397431ef010d","Type":"ContainerDied","Data":"8b3c41a0ba870922f464b0136df69815e7c529ed33852083c132b9a8cd4c239b"} Oct 30 19:34:09.971536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.971514 5733 generic.go:356] "Generic (PLEG): container finished" podID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerID="0925fc42bda40640090a0642ea3a54f725a774de6876917ec90cac8d66765e15" exitCode=0 Oct 30 19:34:09.971617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.971598 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" event={"ID":"bab300d5-80cb-47b0-aacc-0cf91fb86e5f","Type":"ContainerDied","Data":"0925fc42bda40640090a0642ea3a54f725a774de6876917ec90cac8d66765e15"} Oct 30 19:34:09.973248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.973224 5733 generic.go:356] "Generic (PLEG): container finished" podID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerID="a2b50907a0faa4cdc35ebe2e5de045cc97eb4fb4a8280080b8ae71043b5203b3" exitCode=0 Oct 30 19:34:09.973358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.973311 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" event={"ID":"f07707c3-3d89-42b4-a46c-a569fd234f7b","Type":"ContainerDied","Data":"a2b50907a0faa4cdc35ebe2e5de045cc97eb4fb4a8280080b8ae71043b5203b3"} Oct 30 19:34:09.975103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.975080 5733 generic.go:356] "Generic (PLEG): container finished" podID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerID="8978e7df28c56317da669088159c69984c3056f8b7dfc0228b1e5d1b00564342" exitCode=0 Oct 30 19:34:09.975210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.975160 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" event={"ID":"8403f4ea-8e42-41b3-80a1-517d0a8af80b","Type":"ContainerDied","Data":"8978e7df28c56317da669088159c69984c3056f8b7dfc0228b1e5d1b00564342"} Oct 30 19:34:09.977615 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.977444 5733 generic.go:356] "Generic (PLEG): container finished" podID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerID="444017d9ff662e7d9efdb8e594e2aff67213e44183bbd51d811e67ccc17ac73c" exitCode=0 Oct 30 19:34:09.977615 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:09.977474 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" event={"ID":"04482908-bc6a-4e88-9aa3-1090f24e7ebf","Type":"ContainerDied","Data":"444017d9ff662e7d9efdb8e594e2aff67213e44183bbd51d811e67ccc17ac73c"} Oct 30 19:34:10.987800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.987665 5733 generic.go:356] "Generic (PLEG): container finished" podID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerID="8d08ac71e93824066f080274ced1d74a320ae02fcb346fae422ea607ab00e790" exitCode=0 Oct 30 19:34:10.987800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.987716 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" event={"ID":"f07707c3-3d89-42b4-a46c-a569fd234f7b","Type":"ContainerDied","Data":"8d08ac71e93824066f080274ced1d74a320ae02fcb346fae422ea607ab00e790"} Oct 30 19:34:10.990741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.990693 5733 generic.go:356] "Generic (PLEG): container finished" podID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerID="493bf1b4bb04e8e46ffff26eb6ead78e6a7ed6337f45c09dbbcc3fc7e7a52be8" exitCode=0 Oct 30 19:34:10.990825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.990753 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" event={"ID":"4c1363e7-7ee6-4bcc-865e-169343b35123","Type":"ContainerDied","Data":"493bf1b4bb04e8e46ffff26eb6ead78e6a7ed6337f45c09dbbcc3fc7e7a52be8"} Oct 30 19:34:10.995279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.995136 5733 generic.go:356] "Generic (PLEG): container finished" podID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerID="340bf81cd1ab456276a543e33a33f63362d5e5e6d713bdde2643aea1817133c3" exitCode=0 Oct 30 19:34:10.995279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.995221 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" event={"ID":"0980cf99-c017-4662-bca0-c4ea2be970d4","Type":"ContainerDied","Data":"340bf81cd1ab456276a543e33a33f63362d5e5e6d713bdde2643aea1817133c3"} Oct 30 19:34:10.998846 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.998820 5733 generic.go:356] "Generic (PLEG): container finished" podID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerID="58cf1c7979307eb8d15eb24403799a0be7034da47e93d6c72a05e21fe39f2c9e" exitCode=0 Oct 30 19:34:10.998953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:10.998880 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" event={"ID":"8403f4ea-8e42-41b3-80a1-517d0a8af80b","Type":"ContainerDied","Data":"58cf1c7979307eb8d15eb24403799a0be7034da47e93d6c72a05e21fe39f2c9e"} Oct 30 19:34:11.003243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:11.002623 5733 generic.go:356] "Generic (PLEG): container finished" podID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerID="04c3eef06904ffbf4611a23f6284f56080b4d9450b83c601723063644c3708de" exitCode=0 Oct 30 19:34:11.003243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:11.002687 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" event={"ID":"5931d0ca-c1d3-4950-a8b7-b176d2b7e351","Type":"ContainerDied","Data":"04c3eef06904ffbf4611a23f6284f56080b4d9450b83c601723063644c3708de"} Oct 30 19:34:12.017825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:12.017786 5733 generic.go:356] "Generic (PLEG): container finished" podID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerID="22319dac96ced9f8efd140cea31a7262df48631b3bf6ba06034cc55a77a6e5a6" exitCode=0 Oct 30 19:34:12.018331 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:12.017859 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" event={"ID":"5931d0ca-c1d3-4950-a8b7-b176d2b7e351","Type":"ContainerDied","Data":"22319dac96ced9f8efd140cea31a7262df48631b3bf6ba06034cc55a77a6e5a6"} Oct 30 19:34:15.076692 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:15.076650 5733 generic.go:356] "Generic (PLEG): container finished" podID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerID="b6de30fdca31d1d4598225f605364c4bc49ff9a1a58286cddaa7dc1d77580a7d" exitCode=0 Oct 30 19:34:15.077168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:15.076769 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerDied","Data":"b6de30fdca31d1d4598225f605364c4bc49ff9a1a58286cddaa7dc1d77580a7d"} Oct 30 19:34:19.146280 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:19.146199 5733 generic.go:356] "Generic (PLEG): container finished" podID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerID="efae433b09b054eeec266889a490c7f9f731f4f03fd2a6ab585a9d763b807f27" exitCode=0 Oct 30 19:34:19.146749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:19.146342 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" event={"ID":"0980cf99-c017-4662-bca0-c4ea2be970d4","Type":"ContainerDied","Data":"efae433b09b054eeec266889a490c7f9f731f4f03fd2a6ab585a9d763b807f27"} Oct 30 19:34:29.441798 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:34:29.441694 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:34:30.269505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:30.269037 5733 generic.go:356] "Generic (PLEG): container finished" podID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerID="6a64c22221a471b9f0cff8bc4004b24a3fc3d62c1bc82ca03625fd395c7532c5" exitCode=0 Oct 30 19:34:30.269505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:30.269143 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" event={"ID":"04482908-bc6a-4e88-9aa3-1090f24e7ebf","Type":"ContainerDied","Data":"6a64c22221a471b9f0cff8bc4004b24a3fc3d62c1bc82ca03625fd395c7532c5"} Oct 30 19:34:31.281753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.281244 5733 generic.go:356] "Generic (PLEG): container finished" podID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerID="446ff48870285c841ca6313541567253c8522082553e4af44c6ab7d088b3e56d" exitCode=0 Oct 30 19:34:31.281753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.281684 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" event={"ID":"ee56d4bd-99c3-4e04-8e7a-ca5011951c65","Type":"ContainerDied","Data":"446ff48870285c841ca6313541567253c8522082553e4af44c6ab7d088b3e56d"} Oct 30 19:34:31.285799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.285632 5733 generic.go:356] "Generic (PLEG): container finished" podID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerID="790b752131a3338228a184ba9b6b2dd67c99807fa3614d9b036f6b699ee20c9a" exitCode=0 Oct 30 19:34:31.285799 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.285738 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" event={"ID":"7eb4f8fd-3386-4d32-ae7e-397431ef010d","Type":"ContainerDied","Data":"790b752131a3338228a184ba9b6b2dd67c99807fa3614d9b036f6b699ee20c9a"} Oct 30 19:34:31.289415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.288914 5733 generic.go:356] "Generic (PLEG): container finished" podID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerID="92f6db15a883875db0689fa2b48e7a4df896099b441577cd719b8717b394e7f0" exitCode=0 Oct 30 19:34:31.289415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.289329 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" event={"ID":"bab300d5-80cb-47b0-aacc-0cf91fb86e5f","Type":"ContainerDied","Data":"92f6db15a883875db0689fa2b48e7a4df896099b441577cd719b8717b394e7f0"} Oct 30 19:34:31.472910 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.472880 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:34:31.576501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.576408 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\") pod \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " Oct 30 19:34:31.576501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.576480 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\") pod \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " Oct 30 19:34:31.576718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.576560 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\" (UniqueName: \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\") pod \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\" (UID: \"04482908-bc6a-4e88-9aa3-1090f24e7ebf\") " Oct 30 19:34:31.578269 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.578219 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle" pod "04482908-bc6a-4e88-9aa3-1090f24e7ebf" (UID: "04482908-bc6a-4e88-9aa3-1090f24e7ebf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:31.584701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.584676 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util" pod "04482908-bc6a-4e88-9aa3-1090f24e7ebf" (UID: "04482908-bc6a-4e88-9aa3-1090f24e7ebf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:31.590632 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.590589 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr" pod "04482908-bc6a-4e88-9aa3-1090f24e7ebf" (UID: "04482908-bc6a-4e88-9aa3-1090f24e7ebf"). InnerVolumeSpecName "kube-api-access-lvrvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:31.677877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.677833 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:31.678046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.677883 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04482908-bc6a-4e88-9aa3-1090f24e7ebf-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:31.678046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:31.677901 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvrvr\" (UniqueName: \"kubernetes.io/projected/04482908-bc6a-4e88-9aa3-1090f24e7ebf-kube-api-access-lvrvr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.304375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.304323 5733 generic.go:356] "Generic (PLEG): container finished" podID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerID="d0015facd24b6d758469184780c3e29de571a37ada7655745df01e9a12e7788c" exitCode=0 Oct 30 19:34:32.304773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.304352 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" event={"ID":"4c1363e7-7ee6-4bcc-865e-169343b35123","Type":"ContainerDied","Data":"d0015facd24b6d758469184780c3e29de571a37ada7655745df01e9a12e7788c"} Oct 30 19:34:32.306746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.306712 5733 generic.go:356] "Generic (PLEG): container finished" podID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerID="ad39622fa6c401688e56e27c8e9aa9bd0f243068ae8d506707d5b27053fb706b" exitCode=0 Oct 30 19:34:32.306855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.306805 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" event={"ID":"0980cf99-c017-4662-bca0-c4ea2be970d4","Type":"ContainerDied","Data":"ad39622fa6c401688e56e27c8e9aa9bd0f243068ae8d506707d5b27053fb706b"} Oct 30 19:34:32.309114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.309038 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" event={"ID":"04482908-bc6a-4e88-9aa3-1090f24e7ebf","Type":"ContainerDied","Data":"0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62"} Oct 30 19:34:32.309267 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.309251 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd649570fe6a9886ab332073eba0cb4aabefeb1af3906b03ed40b0dc479de62" Oct 30 19:34:32.309378 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.309146 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t6lcs" Oct 30 19:34:32.520245 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.520219 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:34:32.527447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.527421 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:34:32.538440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.538365 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595723 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\") pod \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595783 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\") pod \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595824 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\" (UniqueName: \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\") pod \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\" (UID: \"7eb4f8fd-3386-4d32-ae7e-397431ef010d\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595858 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\") pod \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595935 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\") pod \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.595983 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\" (UniqueName: \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\") pod \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.596014 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\") pod \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\" (UID: \"ee56d4bd-99c3-4e04-8e7a-ca5011951c65\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.596046 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\") pod \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " Oct 30 19:34:32.596743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.596080 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\" (UniqueName: \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\") pod \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\" (UID: \"bab300d5-80cb-47b0-aacc-0cf91fb86e5f\") " Oct 30 19:34:32.597382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.597146 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle" pod "ee56d4bd-99c3-4e04-8e7a-ca5011951c65" (UID: "ee56d4bd-99c3-4e04-8e7a-ca5011951c65"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.601380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.601211 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle" pod "7eb4f8fd-3386-4d32-ae7e-397431ef010d" (UID: "7eb4f8fd-3386-4d32-ae7e-397431ef010d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.601380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.601333 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle" pod "bab300d5-80cb-47b0-aacc-0cf91fb86e5f" (UID: "bab300d5-80cb-47b0-aacc-0cf91fb86e5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.603185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.602559 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n" pod "ee56d4bd-99c3-4e04-8e7a-ca5011951c65" (UID: "ee56d4bd-99c3-4e04-8e7a-ca5011951c65"). InnerVolumeSpecName "kube-api-access-hth9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.606135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.606028 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util" pod "7eb4f8fd-3386-4d32-ae7e-397431ef010d" (UID: "7eb4f8fd-3386-4d32-ae7e-397431ef010d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.607919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.607892 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p" pod "bab300d5-80cb-47b0-aacc-0cf91fb86e5f" (UID: "bab300d5-80cb-47b0-aacc-0cf91fb86e5f"). InnerVolumeSpecName "kube-api-access-sp58p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.609224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.609196 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v" pod "7eb4f8fd-3386-4d32-ae7e-397431ef010d" (UID: "7eb4f8fd-3386-4d32-ae7e-397431ef010d"). InnerVolumeSpecName "kube-api-access-kff5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:32.609433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.609400 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util" pod "ee56d4bd-99c3-4e04-8e7a-ca5011951c65" (UID: "ee56d4bd-99c3-4e04-8e7a-ca5011951c65"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.610609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.610587 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util" pod "bab300d5-80cb-47b0-aacc-0cf91fb86e5f" (UID: "bab300d5-80cb-47b0-aacc-0cf91fb86e5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:32.697244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697208 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hth9n\" (UniqueName: \"kubernetes.io/projected/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-kube-api-access-hth9n\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697239 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697254 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697266 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp58p\" (UniqueName: \"kubernetes.io/projected/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-kube-api-access-sp58p\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697285 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697293 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7eb4f8fd-3386-4d32-ae7e-397431ef010d-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697301 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kff5v\" (UniqueName: \"kubernetes.io/projected/7eb4f8fd-3386-4d32-ae7e-397431ef010d-kube-api-access-kff5v\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697309 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee56d4bd-99c3-4e04-8e7a-ca5011951c65-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:32.697418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:32.697318 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bab300d5-80cb-47b0-aacc-0cf91fb86e5f-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.316413 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.316378 5733 generic.go:356] "Generic (PLEG): container finished" podID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerID="811d98728840ad6935a353f776f0707528f1064df3155652e675e00818e8e386" exitCode=0 Oct 30 19:34:33.316871 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.316459 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" event={"ID":"f07707c3-3d89-42b4-a46c-a569fd234f7b","Type":"ContainerDied","Data":"811d98728840ad6935a353f776f0707528f1064df3155652e675e00818e8e386"} Oct 30 19:34:33.318842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.318812 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerStarted","Data":"2548a2669b720d5e09aaacf2616e0db9df5d561370dc9d5e4453e203fb2eaaa1"} Oct 30 19:34:33.321119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.321078 5733 generic.go:356] "Generic (PLEG): container finished" podID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerID="54bff5eece0efb0371c4aab46968bef9cc3267f45a0846b6656df05e80982980" exitCode=0 Oct 30 19:34:33.321119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.321111 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" event={"ID":"8403f4ea-8e42-41b3-80a1-517d0a8af80b","Type":"ContainerDied","Data":"54bff5eece0efb0371c4aab46968bef9cc3267f45a0846b6656df05e80982980"} Oct 30 19:34:33.323689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.323669 5733 generic.go:356] "Generic (PLEG): container finished" podID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerID="471371035a205af3b9c2b01e2a4b92dc2ee3e179ca499e853bcd21de6eb08142" exitCode=0 Oct 30 19:34:33.323792 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.323775 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" event={"ID":"5931d0ca-c1d3-4950-a8b7-b176d2b7e351","Type":"ContainerDied","Data":"471371035a205af3b9c2b01e2a4b92dc2ee3e179ca499e853bcd21de6eb08142"} Oct 30 19:34:33.325825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.325796 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" event={"ID":"ee56d4bd-99c3-4e04-8e7a-ca5011951c65","Type":"ContainerDied","Data":"37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819"} Oct 30 19:34:33.325825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.325806 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c4xkm5" Oct 30 19:34:33.325987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.325827 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37985a84b87831e390b378973525a1aebee10e0be2b45d48ca6ba7f371454819" Oct 30 19:34:33.328375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.328242 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" Oct 30 19:34:33.328375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.328253 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rc7ww" event={"ID":"7eb4f8fd-3386-4d32-ae7e-397431ef010d","Type":"ContainerDied","Data":"bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303"} Oct 30 19:34:33.328375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.328286 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc20f0aa0f42b9f09fad9bdab6010049a2f5114ccf04cbeeccb25731203ae303" Oct 30 19:34:33.331073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.331046 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" Oct 30 19:34:33.331166 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.331076 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7bmmpf" event={"ID":"bab300d5-80cb-47b0-aacc-0cf91fb86e5f","Type":"ContainerDied","Data":"56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba"} Oct 30 19:34:33.331212 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.331178 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56265dbd3757be6e45165b848b9c88e1332b10e2b929067c2fe95bafa13d68ba" Oct 30 19:34:33.409427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.407855 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" podStartSLOduration=2.252893432 podStartE2EDuration="41.407835094s" totalImagesPullingTime="39.154941662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:33:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:33.405268553 +0000 UTC m=+233.473104133" watchObservedRunningTime="2026-10-30 19:34:33.407835094 +0000 UTC m=+233.475670664" Oct 30 19:34:33.480194 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.480172 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:34:33.502371 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.502345 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:34:33.514273 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514249 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\" (UniqueName: \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\") pod \"0980cf99-c017-4662-bca0-c4ea2be970d4\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " Oct 30 19:34:33.514404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514285 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\") pod \"4c1363e7-7ee6-4bcc-865e-169343b35123\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " Oct 30 19:34:33.514404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514304 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\") pod \"0980cf99-c017-4662-bca0-c4ea2be970d4\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " Oct 30 19:34:33.514404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514377 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\") pod \"0980cf99-c017-4662-bca0-c4ea2be970d4\" (UID: \"0980cf99-c017-4662-bca0-c4ea2be970d4\") " Oct 30 19:34:33.514568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514442 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\" (UniqueName: \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\") pod \"4c1363e7-7ee6-4bcc-865e-169343b35123\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " Oct 30 19:34:33.514568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514500 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\") pod \"4c1363e7-7ee6-4bcc-865e-169343b35123\" (UID: \"4c1363e7-7ee6-4bcc-865e-169343b35123\") " Oct 30 19:34:33.514778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.514756 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle" pod "0980cf99-c017-4662-bca0-c4ea2be970d4" (UID: "0980cf99-c017-4662-bca0-c4ea2be970d4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.515380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.515352 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle" pod "4c1363e7-7ee6-4bcc-865e-169343b35123" (UID: "4c1363e7-7ee6-4bcc-865e-169343b35123"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.516650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.516625 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs" pod "4c1363e7-7ee6-4bcc-865e-169343b35123" (UID: "4c1363e7-7ee6-4bcc-865e-169343b35123"). InnerVolumeSpecName "kube-api-access-6z2zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.516811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.516789 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq" pod "0980cf99-c017-4662-bca0-c4ea2be970d4" (UID: "0980cf99-c017-4662-bca0-c4ea2be970d4"). InnerVolumeSpecName "kube-api-access-c74hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:33.520376 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.520357 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util" pod "0980cf99-c017-4662-bca0-c4ea2be970d4" (UID: "0980cf99-c017-4662-bca0-c4ea2be970d4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.520548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.520528 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util" pod "4c1363e7-7ee6-4bcc-865e-169343b35123" (UID: "4c1363e7-7ee6-4bcc-865e-169343b35123"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:33.615903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615862 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z2zs\" (UniqueName: \"kubernetes.io/projected/4c1363e7-7ee6-4bcc-865e-169343b35123-kube-api-access-6z2zs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.615903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615894 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.615903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615904 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c74hq\" (UniqueName: \"kubernetes.io/projected/0980cf99-c017-4662-bca0-c4ea2be970d4-kube-api-access-c74hq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.615903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615915 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4c1363e7-7ee6-4bcc-865e-169343b35123-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.616202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615924 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.616202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.615932 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0980cf99-c017-4662-bca0-c4ea2be970d4-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:33.854392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:33.854358 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:34.337602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.337507 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" event={"ID":"4c1363e7-7ee6-4bcc-865e-169343b35123","Type":"ContainerDied","Data":"c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504"} Oct 30 19:34:34.337602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.337558 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c814dcc7f33179193336bba26a05ad92b9d1a6299b1299254b2701e88f0e1504" Oct 30 19:34:34.337602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.337524 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a4c99f" Oct 30 19:34:34.339824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.339794 5733 generic.go:356] "Generic (PLEG): container finished" podID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerID="2548a2669b720d5e09aaacf2616e0db9df5d561370dc9d5e4453e203fb2eaaa1" exitCode=0 Oct 30 19:34:34.339976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.339888 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerDied","Data":"2548a2669b720d5e09aaacf2616e0db9df5d561370dc9d5e4453e203fb2eaaa1"} Oct 30 19:34:34.341739 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.341720 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" Oct 30 19:34:34.341865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.341755 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644x4wft" event={"ID":"0980cf99-c017-4662-bca0-c4ea2be970d4","Type":"ContainerDied","Data":"7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3"} Oct 30 19:34:34.341865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.341792 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb79905cb84e4eeab41b6b7c1639c86db2a360a9aed1a476e629b2979fc2ac3" Oct 30 19:34:34.483822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.483793 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:34:34.524940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.524917 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\" (UniqueName: \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\") pod \"f07707c3-3d89-42b4-a46c-a569fd234f7b\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " Oct 30 19:34:34.525073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.524968 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\") pod \"f07707c3-3d89-42b4-a46c-a569fd234f7b\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " Oct 30 19:34:34.525073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.525044 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\") pod \"f07707c3-3d89-42b4-a46c-a569fd234f7b\" (UID: \"f07707c3-3d89-42b4-a46c-a569fd234f7b\") " Oct 30 19:34:34.525604 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.525579 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle" pod "f07707c3-3d89-42b4-a46c-a569fd234f7b" (UID: "f07707c3-3d89-42b4-a46c-a569fd234f7b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.526928 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.526902 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq" pod "f07707c3-3d89-42b4-a46c-a569fd234f7b" (UID: "f07707c3-3d89-42b4-a46c-a569fd234f7b"). InnerVolumeSpecName "kube-api-access-xwghq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:34.527047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.526956 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:34:34.534104 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.534071 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util" pod "f07707c3-3d89-42b4-a46c-a569fd234f7b" (UID: "f07707c3-3d89-42b4-a46c-a569fd234f7b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.537526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.537504 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:34:34.626007 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.625979 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\" (UniqueName: \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\") pod \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " Oct 30 19:34:34.626178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626076 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\") pod \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " Oct 30 19:34:34.626178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626156 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\") pod \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " Oct 30 19:34:34.626266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626195 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\") pod \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\" (UID: \"8403f4ea-8e42-41b3-80a1-517d0a8af80b\") " Oct 30 19:34:34.626266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626236 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\") pod \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " Oct 30 19:34:34.626350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626263 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\" (UniqueName: \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\") pod \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\" (UID: \"5931d0ca-c1d3-4950-a8b7-b176d2b7e351\") " Oct 30 19:34:34.626535 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626515 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.626535 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626538 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f07707c3-3d89-42b4-a46c-a569fd234f7b-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.626693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626555 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwghq\" (UniqueName: \"kubernetes.io/projected/f07707c3-3d89-42b4-a46c-a569fd234f7b-kube-api-access-xwghq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.626787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626758 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle" pod "5931d0ca-c1d3-4950-a8b7-b176d2b7e351" (UID: "5931d0ca-c1d3-4950-a8b7-b176d2b7e351"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.626955 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.626936 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle" pod "8403f4ea-8e42-41b3-80a1-517d0a8af80b" (UID: "8403f4ea-8e42-41b3-80a1-517d0a8af80b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.628572 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.628556 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j" pod "8403f4ea-8e42-41b3-80a1-517d0a8af80b" (UID: "8403f4ea-8e42-41b3-80a1-517d0a8af80b"). InnerVolumeSpecName "kube-api-access-6qh4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:34.628770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.628748 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88" pod "5931d0ca-c1d3-4950-a8b7-b176d2b7e351" (UID: "5931d0ca-c1d3-4950-a8b7-b176d2b7e351"). InnerVolumeSpecName "kube-api-access-84t88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:34.632493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.632471 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util" pod "8403f4ea-8e42-41b3-80a1-517d0a8af80b" (UID: "8403f4ea-8e42-41b3-80a1-517d0a8af80b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.632547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.632476 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util" pod "5931d0ca-c1d3-4950-a8b7-b176d2b7e351" (UID: "5931d0ca-c1d3-4950-a8b7-b176d2b7e351"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:34.727550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727518 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.727550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727541 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.727550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727550 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8403f4ea-8e42-41b3-80a1-517d0a8af80b-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.727550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727558 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.727785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727567 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84t88\" (UniqueName: \"kubernetes.io/projected/5931d0ca-c1d3-4950-a8b7-b176d2b7e351-kube-api-access-84t88\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.727785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.727576 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qh4j\" (UniqueName: \"kubernetes.io/projected/8403f4ea-8e42-41b3-80a1-517d0a8af80b-kube-api-access-6qh4j\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:34.832528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832496 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg"] Oct 30 19:34:34.832961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832906 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="util" Oct 30 19:34:34.832961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832928 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="util" Oct 30 19:34:34.832961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832946 5733 cpu_manager.go:414] "containerMap: removing container" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="pull" Oct 30 19:34:34.832961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832953 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="pull" Oct 30 19:34:34.832961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832965 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832972 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832984 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832990 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.832999 5733 cpu_manager.go:414] "containerMap: removing container" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833005 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833015 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833023 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833038 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833044 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833067 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833074 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833093 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833101 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833111 5733 cpu_manager.go:414] "containerMap: removing container" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833117 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833126 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833133 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833149 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833155 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833170 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833176 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833187 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833193 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833202 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833209 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833220 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833227 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833239 5733 cpu_manager.go:414] "containerMap: removing container" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833246 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833256 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833262 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833278 5733 cpu_manager.go:414] "containerMap: removing container" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833285 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833297 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833304 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="extract" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833320 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833326 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833335 5733 cpu_manager.go:414] "containerMap: removing container" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833342 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833354 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833361 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833370 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833376 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833385 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833392 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833401 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833408 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="util" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833419 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="pull" Oct 30 19:34:34.833449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833425 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="pull" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833502 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8403f4ea-8e42-41b3-80a1-517d0a8af80b" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833515 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5931d0ca-c1d3-4950-a8b7-b176d2b7e351" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833525 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab300d5-80cb-47b0-aacc-0cf91fb86e5f" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833534 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c1363e7-7ee6-4bcc-865e-169343b35123" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833542 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee56d4bd-99c3-4e04-8e7a-ca5011951c65" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833550 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07707c3-3d89-42b4-a46c-a569fd234f7b" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833557 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="04482908-bc6a-4e88-9aa3-1090f24e7ebf" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833565 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb4f8fd-3386-4d32-ae7e-397431ef010d" containerName="extract" Oct 30 19:34:34.835159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.833574 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0980cf99-c017-4662-bca0-c4ea2be970d4" containerName="extract" Oct 30 19:34:34.836313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.836297 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.842287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.842268 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:34:34.845530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.845512 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:34:34.852699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.852682 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:34:34.876908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.876861 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg"] Oct 30 19:34:34.929000 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.928977 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.929119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.929021 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.929119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.929051 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.929197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.929150 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.929197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.929184 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkdfp\" (UniqueName: \"kubernetes.io/projected/82727e18-39c9-438c-b335-259eb7281109-kube-api-access-zkdfp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.929254 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.929204 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:34.942603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.942582 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:34:34.942738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.942727 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:34:34.942985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:34.942967 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-klkgs\"" Oct 30 19:34:35.030358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030326 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030369 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkdfp\" (UniqueName: \"kubernetes.io/projected/82727e18-39c9-438c-b335-259eb7281109-kube-api-access-zkdfp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030400 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030452 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030501 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030539 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.030871 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.030851 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-manifests\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.032741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.032718 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82727e18-39c9-438c-b335-259eb7281109-tmp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.032944 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.032929 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.032987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.032958 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-webhook-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.033018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.033004 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82727e18-39c9-438c-b335-259eb7281109-metrics-cert\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.050073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.050039 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkdfp\" (UniqueName: \"kubernetes.io/projected/82727e18-39c9-438c-b335-259eb7281109-kube-api-access-zkdfp\") pod \"opendatahub-operator-controller-manager-85768f955d-hxzvg\" (UID: \"82727e18-39c9-438c-b335-259eb7281109\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.149406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.149344 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:35.348048 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.348009 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" event={"ID":"8403f4ea-8e42-41b3-80a1-517d0a8af80b","Type":"ContainerDied","Data":"c7e75b3c4b3dc9004f0f64b8775700fb69f2654174622b50d3393484f4ba9771"} Oct 30 19:34:35.348481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.348094 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7e75b3c4b3dc9004f0f64b8775700fb69f2654174622b50d3393484f4ba9771" Oct 30 19:34:35.348481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.348020 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axz657" Oct 30 19:34:35.350106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.350085 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" event={"ID":"5931d0ca-c1d3-4950-a8b7-b176d2b7e351","Type":"ContainerDied","Data":"84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d"} Oct 30 19:34:35.350240 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.350112 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7d5s8r" Oct 30 19:34:35.350240 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.350120 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84821d55cfed1ae439af28fc2c43dd2e4538c8b364067b0c633613f0c410710d" Oct 30 19:34:35.352183 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.352154 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" event={"ID":"f07707c3-3d89-42b4-a46c-a569fd234f7b","Type":"ContainerDied","Data":"24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8"} Oct 30 19:34:35.352295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.352198 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24291fdb7ad221112466860efca77cdc9c0fe1bc6f7b7dfd17dd400da76124e8" Oct 30 19:34:35.352295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.352199 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czvtl9" Oct 30 19:34:35.457550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.457512 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg"] Oct 30 19:34:35.460071 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:34:35.460031 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82727e18_39c9_438c_b335_259eb7281109.slice/crio-a64261ab99c842d6f71c1dd7bc5d9d24846750b89087aa592fe96efcec29ac39 WatchSource:0}: Error finding container a64261ab99c842d6f71c1dd7bc5d9d24846750b89087aa592fe96efcec29ac39: Status 404 returned error can't find the container with id a64261ab99c842d6f71c1dd7bc5d9d24846750b89087aa592fe96efcec29ac39 Oct 30 19:34:35.465349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.465332 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:34:35.534888 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.534853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\") pod \"4559fb0d-c4cc-444f-998c-176d280d8efa\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " Oct 30 19:34:35.535034 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.534899 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\" (UniqueName: \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\") pod \"4559fb0d-c4cc-444f-998c-176d280d8efa\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " Oct 30 19:34:35.535034 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.534963 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\") pod \"4559fb0d-c4cc-444f-998c-176d280d8efa\" (UID: \"4559fb0d-c4cc-444f-998c-176d280d8efa\") " Oct 30 19:34:35.535711 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.535680 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle" pod "4559fb0d-c4cc-444f-998c-176d280d8efa" (UID: "4559fb0d-c4cc-444f-998c-176d280d8efa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:35.537123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.537095 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx" pod "4559fb0d-c4cc-444f-998c-176d280d8efa" (UID: "4559fb0d-c4cc-444f-998c-176d280d8efa"). InnerVolumeSpecName "kube-api-access-4t4lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:34:35.540912 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.540892 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util" pod "4559fb0d-c4cc-444f-998c-176d280d8efa" (UID: "4559fb0d-c4cc-444f-998c-176d280d8efa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:34:35.636443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.636403 5733 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-bundle\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:35.636443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.636434 5733 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4559fb0d-c4cc-444f-998c-176d280d8efa-util\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:35.636443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.636445 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t4lx\" (UniqueName: \"kubernetes.io/projected/4559fb0d-c4cc-444f-998c-176d280d8efa-kube-api-access-4t4lx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:34:35.759325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:35.759233 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:36.143005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.142945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:36.296948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.296910 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:36.362870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.362242 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" Oct 30 19:34:36.362870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.362251 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651crf7hb" event={"ID":"4559fb0d-c4cc-444f-998c-176d280d8efa","Type":"ContainerDied","Data":"acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90"} Oct 30 19:34:36.362870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.362303 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acd08b7b6abbfea187775316ec651c4228cc47c383c295a12ce917c1f6c45e90" Oct 30 19:34:36.364775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:36.364750 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerStarted","Data":"a64261ab99c842d6f71c1dd7bc5d9d24846750b89087aa592fe96efcec29ac39"} Oct 30 19:34:38.324510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:38.324480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:38.376154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:38.376107 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerStarted","Data":"335e149a379e86fce441bb5f033ed5ca5098ca13f16f26ff3119d2f4ce0a1fb1"} Oct 30 19:34:38.484918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:38.484843 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:38.630393 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:38.630359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:39.381940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:39.381908 5733 generic.go:356] "Generic (PLEG): container finished" podID="82727e18-39c9-438c-b335-259eb7281109" containerID="335e149a379e86fce441bb5f033ed5ca5098ca13f16f26ff3119d2f4ce0a1fb1" exitCode=0 Oct 30 19:34:39.382416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:39.381981 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerDied","Data":"335e149a379e86fce441bb5f033ed5ca5098ca13f16f26ff3119d2f4ce0a1fb1"} Oct 30 19:34:40.387913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.387879 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerStarted","Data":"b2a69ffb4d46878be72aef5733be04704c160667c3017bb4249856965774fc45"} Oct 30 19:34:40.388297 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.387968 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:40.442003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.441919 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podStartSLOduration=2.779647031 podStartE2EDuration="6.441901284s" totalImagesPullingTime="3.160989271s" totalInitContainerRuntime="501.264982ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:40.43818213 +0000 UTC m=+240.506017703" watchObservedRunningTime="2026-10-30 19:34:40.441901284 +0000 UTC m=+240.509736852" Oct 30 19:34:40.919102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919070 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z"] Oct 30 19:34:40.919364 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919352 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="util" Oct 30 19:34:40.919419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919365 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="util" Oct 30 19:34:40.919419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919391 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="extract" Oct 30 19:34:40.919419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919396 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="extract" Oct 30 19:34:40.919419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919405 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="pull" Oct 30 19:34:40.919419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919410 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="pull" Oct 30 19:34:40.919590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.919454 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4559fb0d-c4cc-444f-998c-176d280d8efa" containerName="extract" Oct 30 19:34:40.921979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.921962 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:40.942901 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.942878 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hg2fd\"" Oct 30 19:34:40.943417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.943396 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z"] Oct 30 19:34:40.954973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.954950 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:34:40.955157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.955144 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:34:40.986743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.985881 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8bac1e-c263-46f7-97e7-860f85904afc-tmp\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:40.986743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:40.985938 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28nv\" (UniqueName: \"kubernetes.io/projected/7a8bac1e-c263-46f7-97e7-860f85904afc-kube-api-access-q28nv\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.086920 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.086887 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8bac1e-c263-46f7-97e7-860f85904afc-tmp\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.087116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.086928 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28nv\" (UniqueName: \"kubernetes.io/projected/7a8bac1e-c263-46f7-97e7-860f85904afc-kube-api-access-q28nv\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.087300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.087278 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a8bac1e-c263-46f7-97e7-860f85904afc-tmp\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.101522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.101496 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28nv\" (UniqueName: \"kubernetes.io/projected/7a8bac1e-c263-46f7-97e7-860f85904afc-kube-api-access-q28nv\") pod \"jobset-operator-674cd59dc5-dwm4z\" (UID: \"7a8bac1e-c263-46f7-97e7-860f85904afc\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.244876 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.244810 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" Oct 30 19:34:41.525052 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:41.525027 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z"] Oct 30 19:34:41.527583 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:34:41.527554 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a8bac1e_c263_46f7_97e7_860f85904afc.slice/crio-f1bcbb03fcfd0c1bb7c9072dabee013f8f731d73c76bd1773df55d8ebf058b33 WatchSource:0}: Error finding container f1bcbb03fcfd0c1bb7c9072dabee013f8f731d73c76bd1773df55d8ebf058b33: Status 404 returned error can't find the container with id f1bcbb03fcfd0c1bb7c9072dabee013f8f731d73c76bd1773df55d8ebf058b33 Oct 30 19:34:42.399504 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:42.399470 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" event={"ID":"7a8bac1e-c263-46f7-97e7-860f85904afc","Type":"ContainerStarted","Data":"f1bcbb03fcfd0c1bb7c9072dabee013f8f731d73c76bd1773df55d8ebf058b33"} Oct 30 19:34:43.873768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:43.873733 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.033163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:44.033079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:44.410655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:44.410620 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" event={"ID":"7a8bac1e-c263-46f7-97e7-860f85904afc","Type":"ContainerStarted","Data":"44a4c38d245126415339bcb71dee824220c205612fbe363b50b3bb44f836ebe1"} Oct 30 19:34:44.432673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:44.432628 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" podStartSLOduration=2.317081804 podStartE2EDuration="4.432616704s" totalImagesPullingTime="2.1155349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:44.432297118 +0000 UTC m=+244.500132687" watchObservedRunningTime="2026-10-30 19:34:44.432616704 +0000 UTC m=+244.500452271" Oct 30 19:34:49.453564 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:49.453530 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:49.870630 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:49.870597 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:51.396499 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:51.396468 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 19:34:52.645110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:52.645076 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:54.658842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:54.658802 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:57.303175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.303143 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5"] Oct 30 19:34:57.306615 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.306596 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:34:57.322178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.322158 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-shlrr\"" Oct 30 19:34:57.323384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.323360 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:34:57.325678 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.325653 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5"] Oct 30 19:34:57.326718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.326699 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:34:57.423520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.423487 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 19:34:57.435041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.435015 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2glbr\" (UniqueName: \"kubernetes.io/projected/56747b0a-051b-4e92-a51a-b71b394a2de2-kube-api-access-2glbr\") pod \"dns-operator-controller-manager-8557cbdb84-2s5r5\" (UID: \"56747b0a-051b-4e92-a51a-b71b394a2de2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:34:57.536216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.536186 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2glbr\" (UniqueName: \"kubernetes.io/projected/56747b0a-051b-4e92-a51a-b71b394a2de2-kube-api-access-2glbr\") pod \"dns-operator-controller-manager-8557cbdb84-2s5r5\" (UID: \"56747b0a-051b-4e92-a51a-b71b394a2de2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:34:57.554614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.554548 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2glbr\" (UniqueName: \"kubernetes.io/projected/56747b0a-051b-4e92-a51a-b71b394a2de2-kube-api-access-2glbr\") pod \"dns-operator-controller-manager-8557cbdb84-2s5r5\" (UID: \"56747b0a-051b-4e92-a51a-b71b394a2de2\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:34:57.619474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.619449 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:34:57.904447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:57.904424 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5"] Oct 30 19:34:57.906409 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:34:57.906381 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56747b0a_051b_4e92_a51a_b71b394a2de2.slice/crio-5db8e3727f8ee2d84151f0eb8392870c12dc5de0f86dc86eb5e5a5c9210e0bf2 WatchSource:0}: Error finding container 5db8e3727f8ee2d84151f0eb8392870c12dc5de0f86dc86eb5e5a5c9210e0bf2: Status 404 returned error can't find the container with id 5db8e3727f8ee2d84151f0eb8392870c12dc5de0f86dc86eb5e5a5c9210e0bf2 Oct 30 19:34:58.045834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:58.045802 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:58.473993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:58.473958 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" event={"ID":"56747b0a-051b-4e92-a51a-b71b394a2de2","Type":"ContainerStarted","Data":"5db8e3727f8ee2d84151f0eb8392870c12dc5de0f86dc86eb5e5a5c9210e0bf2"} Oct 30 19:34:58.512704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:58.512677 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:58.655371 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:58.655337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:58.919050 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:58.919008 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.062100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:59.062066 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.231907 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:59.231810 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:59.387935 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:34:59.387896 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:34:59.664251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:34:59.664220 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:00.884495 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:00.884447 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:01.490131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:01.490095 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" event={"ID":"56747b0a-051b-4e92-a51a-b71b394a2de2","Type":"ContainerStarted","Data":"8f6f1a7ab236f09929bdcaeca86bf70d85d8bedf8c2b54a400acc867f408a7e7"} Oct 30 19:35:01.490321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:01.490193 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:35:01.512303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:01.512253 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podStartSLOduration=1.368230201 podStartE2EDuration="4.512240866s" totalImagesPullingTime="3.144010665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:01.511334439 +0000 UTC m=+261.579170008" watchObservedRunningTime="2026-10-30 19:35:01.512240866 +0000 UTC m=+261.580076433" Oct 30 19:35:02.332335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:02.332283 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:02.508948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:02.508902 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.351801 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.351767 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:35:04.354094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.354075 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.362425 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.362398 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kjn7n\"" Oct 30 19:35:04.373212 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.373185 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:35:04.495842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.495805 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.496022 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.495923 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvft\" (UniqueName: \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.597080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.597013 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvft\" (UniqueName: \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.597293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.597131 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.597604 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.597579 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.611370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.611304 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvft\" (UniqueName: \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\") pod \"kuadrant-operator-controller-manager-678d87766f-j7pn2\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.666298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.666267 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:04.871749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.871659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:04.952853 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:04.952814 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:35:04.961214 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:35:04.961183 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d1616c_adc0_4619_93f9_b75ca7d6568b.slice/crio-5a2cdf74ccc7f9076b62859d08a7e882ac48d1da7837c5eadfb16293221f7750 WatchSource:0}: Error finding container 5a2cdf74ccc7f9076b62859d08a7e882ac48d1da7837c5eadfb16293221f7750: Status 404 returned error can't find the container with id 5a2cdf74ccc7f9076b62859d08a7e882ac48d1da7837c5eadfb16293221f7750 Oct 30 19:35:05.510351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:05.510310 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" event={"ID":"27d1616c-adc0-4619-93f9-b75ca7d6568b","Type":"ContainerStarted","Data":"5a2cdf74ccc7f9076b62859d08a7e882ac48d1da7837c5eadfb16293221f7750"} Oct 30 19:35:05.611445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:05.611409 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:05.781276 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:05.781186 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.473555 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:08.473519 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:08.619669 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:08.619631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.670883 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.670849 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.846144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.846100 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:10.974781 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.974699 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x4v2z"] Oct 30 19:35:10.977483 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.977451 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:10.987384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.987362 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 19:35:10.987384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.987370 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 19:35:10.988936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.988916 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:35:10.989028 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.988976 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:35:10.998464 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:10.998424 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x4v2z"] Oct 30 19:35:11.076320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.076267 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.087944 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.087919 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqw8g\"" Oct 30 19:35:11.151376 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.151346 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff194676-b322-42b3-a970-7db28cc2ac3c-tls-certs\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.151672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.151467 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff194676-b322-42b3-a970-7db28cc2ac3c-tmp\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.151672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.151629 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjn6s\" (UniqueName: \"kubernetes.io/projected/ff194676-b322-42b3-a970-7db28cc2ac3c-kube-api-access-xjn6s\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.261597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.261510 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff194676-b322-42b3-a970-7db28cc2ac3c-tmp\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.261597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.261557 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjn6s\" (UniqueName: \"kubernetes.io/projected/ff194676-b322-42b3-a970-7db28cc2ac3c-kube-api-access-xjn6s\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.261828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.261637 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff194676-b322-42b3-a970-7db28cc2ac3c-tls-certs\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.264759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.264529 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ff194676-b322-42b3-a970-7db28cc2ac3c-tmp\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.264759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.264709 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff194676-b322-42b3-a970-7db28cc2ac3c-tls-certs\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.292552 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.292493 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjn6s\" (UniqueName: \"kubernetes.io/projected/ff194676-b322-42b3-a970-7db28cc2ac3c-kube-api-access-xjn6s\") pod \"kube-auth-proxy-5d87896589-x4v2z\" (UID: \"ff194676-b322-42b3-a970-7db28cc2ac3c\") " pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.309103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.309052 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.551216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.551108 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.592453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.592426 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" Oct 30 19:35:11.867319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.867283 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:11.887789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:11.887760 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d87896589-x4v2z"] Oct 30 19:35:11.889553 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:35:11.889522 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff194676_b322_42b3_a970_7db28cc2ac3c.slice/crio-dc5c5dda3d0b77c640c694e1c2c8d8f80690ecf9f0e306ddfb94a52597d31fe6 WatchSource:0}: Error finding container dc5c5dda3d0b77c640c694e1c2c8d8f80690ecf9f0e306ddfb94a52597d31fe6: Status 404 returned error can't find the container with id dc5c5dda3d0b77c640c694e1c2c8d8f80690ecf9f0e306ddfb94a52597d31fe6 Oct 30 19:35:12.071645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:12.071609 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:12.497624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:12.497590 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 19:35:12.546242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:12.546170 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" event={"ID":"ff194676-b322-42b3-a970-7db28cc2ac3c","Type":"ContainerStarted","Data":"dc5c5dda3d0b77c640c694e1c2c8d8f80690ecf9f0e306ddfb94a52597d31fe6"} Oct 30 19:35:12.619559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:12.619504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:13.178139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:13.178026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:13.539699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:13.539581 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:13.696511 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:13.696448 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:14.163037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.163004 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-glbct"] Oct 30 19:35:14.173993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.173967 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.178924 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.178893 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-glbct"] Oct 30 19:35:14.182549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.182509 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 19:35:14.184927 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.184905 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 19:35:14.201090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.199667 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4czw\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-kube-api-access-g4czw\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.201090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.199746 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.215420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.215390 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jcv8l\"" Oct 30 19:35:14.301087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.300908 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.301087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.300999 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4czw\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-kube-api-access-g4czw\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.319919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.319880 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.320859 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.320812 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4czw\" (UniqueName: \"kubernetes.io/projected/3bb9e4d1-26a1-4698-9322-d963463664b6-kube-api-access-g4czw\") pod \"cert-manager-webhook-5dbbbf75cc-glbct\" (UID: \"3bb9e4d1-26a1-4698-9322-d963463664b6\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.492863 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.492784 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:14.806265 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:14.806240 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-glbct"] Oct 30 19:35:14.809024 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:35:14.808992 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bb9e4d1_26a1_4698_9322_d963463664b6.slice/crio-f75ae8b28bfec25757d8c3b00e455e7b91197dd32ecfc4672698d275619fbb3a WatchSource:0}: Error finding container f75ae8b28bfec25757d8c3b00e455e7b91197dd32ecfc4672698d275619fbb3a: Status 404 returned error can't find the container with id f75ae8b28bfec25757d8c3b00e455e7b91197dd32ecfc4672698d275619fbb3a Oct 30 19:35:15.569544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.569509 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" event={"ID":"27d1616c-adc0-4619-93f9-b75ca7d6568b","Type":"ContainerStarted","Data":"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae"} Oct 30 19:35:15.570004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.569604 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:15.570822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.570797 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" event={"ID":"3bb9e4d1-26a1-4698-9322-d963463664b6","Type":"ContainerStarted","Data":"f75ae8b28bfec25757d8c3b00e455e7b91197dd32ecfc4672698d275619fbb3a"} Oct 30 19:35:15.572224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.572201 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" event={"ID":"ff194676-b322-42b3-a970-7db28cc2ac3c","Type":"ContainerStarted","Data":"2cc72dde64982a96bb0a7eaf3b7c9e18645df970fec695bf6e49cc36aef1d638"} Oct 30 19:35:15.594958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.594916 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" podStartSLOduration=2.01507087 podStartE2EDuration="11.594905942s" totalImagesPullingTime="9.579835072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:15.594104656 +0000 UTC m=+275.661940220" watchObservedRunningTime="2026-10-30 19:35:15.594905942 +0000 UTC m=+275.662741509" Oct 30 19:35:15.616197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.616151 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d87896589-x4v2z" podStartSLOduration=2.771663469 podStartE2EDuration="5.616139995s" totalImagesPullingTime="2.844476526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:15.615744946 +0000 UTC m=+275.683580514" watchObservedRunningTime="2026-10-30 19:35:15.616139995 +0000 UTC m=+275.683975563" Oct 30 19:35:15.915728 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:15.915695 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.118335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:16.118295 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:16.863832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:16.863799 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:17.042232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:17.042198 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:20.599002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:20.598961 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" event={"ID":"3bb9e4d1-26a1-4698-9322-d963463664b6","Type":"ContainerStarted","Data":"0b4442e6aa1a879be1b2fb00cc6bb681c4c200ce506718e1a3e063cdea0966e1"} Oct 30 19:35:20.599469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:20.599092 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:20.620357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:20.620311 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podStartSLOduration=1.438149747 podStartE2EDuration="6.620298311s" totalImagesPullingTime="5.182148564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:20.619230737 +0000 UTC m=+280.687066304" watchObservedRunningTime="2026-10-30 19:35:20.620298311 +0000 UTC m=+280.688133879" Oct 30 19:35:21.357950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:21.357911 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:21.509712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:21.509680 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.262777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:22.262743 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.396484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:22.396454 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.536373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:22.536291 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:22.671368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:22.671338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:23.511004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:23.510969 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:24.842585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.842554 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-ngwls"] Oct 30 19:35:24.845935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.845919 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:24.852163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.852140 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gr7dw\"" Oct 30 19:35:24.860228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.860164 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ngwls"] Oct 30 19:35:24.896457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.896429 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66ccr\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-kube-api-access-66ccr\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:24.896592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.896495 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-bound-sa-token\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:24.997843 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.997802 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66ccr\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-kube-api-access-66ccr\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:24.998011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:24.997906 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-bound-sa-token\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:25.013866 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.013828 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66ccr\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-kube-api-access-66ccr\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:25.013999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.013828 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/63e2b306-7302-41be-92fa-0c0dbeb0e6be-bound-sa-token\") pod \"cert-manager-597b95cb84-ngwls\" (UID: \"63e2b306-7302-41be-92fa-0c0dbeb0e6be\") " pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:25.157507 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.157468 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-ngwls" Oct 30 19:35:25.440513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.440395 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-ngwls"] Oct 30 19:35:25.442933 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:35:25.442902 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63e2b306_7302_41be_92fa_0c0dbeb0e6be.slice/crio-ccc8c85aa7ad26ba6bfc24012ed75c1b6bc1697cac3c49fde8cca2152b5f7126 WatchSource:0}: Error finding container ccc8c85aa7ad26ba6bfc24012ed75c1b6bc1697cac3c49fde8cca2152b5f7126: Status 404 returned error can't find the container with id ccc8c85aa7ad26ba6bfc24012ed75c1b6bc1697cac3c49fde8cca2152b5f7126 Oct 30 19:35:25.624650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.624610 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ngwls" event={"ID":"63e2b306-7302-41be-92fa-0c0dbeb0e6be","Type":"ContainerStarted","Data":"4f257ca07bdf7398a5bb13ac8a9536ce12f47e04fbf9f559968d4ac6999c4099"} Oct 30 19:35:25.624817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.624664 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-ngwls" event={"ID":"63e2b306-7302-41be-92fa-0c0dbeb0e6be","Type":"ContainerStarted","Data":"ccc8c85aa7ad26ba6bfc24012ed75c1b6bc1697cac3c49fde8cca2152b5f7126"} Oct 30 19:35:25.645300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:25.645253 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-ngwls" podStartSLOduration=1.645240645 podStartE2EDuration="1.645240645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:25.644796946 +0000 UTC m=+285.712632515" watchObservedRunningTime="2026-10-30 19:35:25.645240645 +0000 UTC m=+285.713076212" Oct 30 19:35:26.580112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:26.580082 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:35:26.605612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:26.605578 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 19:35:29.375387 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:35:29.375349 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:35:30.435888 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:30.435854 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.646712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:30.646677 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:33.043108 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:33.043073 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:33.223585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:33.223550 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.756931 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:35.756898 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.919991 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:35.919956 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:40.760853 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:40.760816 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:40.913128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:40.913076 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.116355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:41.116315 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.318200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:41.318161 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:41.742016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:41.741979 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:42.002979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.002896 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:42.441852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.441819 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh"] Oct 30 19:35:42.448136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.448111 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.459917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.459893 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh"] Oct 30 19:35:42.466765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.466745 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 19:35:42.466919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.466903 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 19:35:42.466983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.466918 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f78dn\"" Oct 30 19:35:42.468207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.468192 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 19:35:42.545649 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.545617 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-metrics-certs\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.545800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.545657 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgl56\" (UniqueName: \"kubernetes.io/projected/03223b13-85fc-4dda-bf3b-dec632a94c83-kube-api-access-pgl56\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.545800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.545782 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03223b13-85fc-4dda-bf3b-dec632a94c83-manager-config\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.545919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.545825 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-cert\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.646527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.646486 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgl56\" (UniqueName: \"kubernetes.io/projected/03223b13-85fc-4dda-bf3b-dec632a94c83-kube-api-access-pgl56\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.646720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.646555 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03223b13-85fc-4dda-bf3b-dec632a94c83-manager-config\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.646720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.646592 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-cert\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.646720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.646657 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-metrics-certs\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.647399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.647373 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/03223b13-85fc-4dda-bf3b-dec632a94c83-manager-config\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.649220 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.649193 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-cert\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.649312 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.649231 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03223b13-85fc-4dda-bf3b-dec632a94c83-metrics-certs\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.660148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.660126 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgl56\" (UniqueName: \"kubernetes.io/projected/03223b13-85fc-4dda-bf3b-dec632a94c83-kube-api-access-pgl56\") pod \"jobset-controller-manager-64d7d576f9-fb9qh\" (UID: \"03223b13-85fc-4dda-bf3b-dec632a94c83\") " pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:42.761030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:42.760968 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:43.035218 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:43.035193 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh"] Oct 30 19:35:43.036725 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:35:43.036690 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03223b13_85fc_4dda_bf3b_dec632a94c83.slice/crio-f30946425ab07279300de5f1dc1facdc5941179bd1aff7c67bf9d84d846fecb7 WatchSource:0}: Error finding container f30946425ab07279300de5f1dc1facdc5941179bd1aff7c67bf9d84d846fecb7: Status 404 returned error can't find the container with id f30946425ab07279300de5f1dc1facdc5941179bd1aff7c67bf9d84d846fecb7 Oct 30 19:35:43.713367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:43.713332 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" event={"ID":"03223b13-85fc-4dda-bf3b-dec632a94c83","Type":"ContainerStarted","Data":"f30946425ab07279300de5f1dc1facdc5941179bd1aff7c67bf9d84d846fecb7"} Oct 30 19:35:46.729902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:46.729868 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" event={"ID":"03223b13-85fc-4dda-bf3b-dec632a94c83","Type":"ContainerStarted","Data":"3e71065b395cfface84d2648874d38eb412a42167e2ebc5fdef122321694ef0e"} Oct 30 19:35:46.730299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:46.729972 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:46.752539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:46.752495 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podStartSLOduration=1.161928159 podStartE2EDuration="4.752480474s" totalImagesPullingTime="3.590552315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:46.75133956 +0000 UTC m=+306.819175130" watchObservedRunningTime="2026-10-30 19:35:46.752480474 +0000 UTC m=+306.820316042" Oct 30 19:35:49.518119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:49.518087 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.707303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:49.707251 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.740139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:35:57.740106 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 19:35:59.375840 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:35:59.375793 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:36:10.247014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:10.246967 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:10.430223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:10.430175 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:10.619821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:10.619768 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:10.777807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:10.777778 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.922901 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:11.922866 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:12.071329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:12.071293 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:12.216361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:12.216278 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:15.711418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:15.711387 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:29.375314 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:36:29.375270 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:36:49.540586 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:36:49.540550 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:59.375504 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:36:59.375427 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:37:29.375991 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:37:29.375949 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:37:49.590396 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:37:49.590362 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:59.375649 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:37:59.375605 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:38:03.640259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:38:03.640217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:29.376081 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:38:29.375977 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:38:49.647275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:38:49.647229 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:59.375537 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:38:59.375495 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:39:00.419241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:39:00.419211 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:01.294325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:39:01.294288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:29.375665 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:39:29.375624 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:39:49.726268 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:39:49.726212 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:53.913107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:39:53.913003 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:54.071232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:39:54.071201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:59.375565 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:39:59.375532 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:40:29.375367 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:40:29.375330 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:40:49.787628 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:40:49.787590 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.375784 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:40:59.375747 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:41:25.835048 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:41:25.834966 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:25.981198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:41:25.981165 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.376341 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:41:29.376298 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:41:49.822004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:41:49.821971 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.376076 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:41:59.376029 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:42:29.375681 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:42:29.375647 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:42:49.879390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:42:49.879336 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.375614 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:42:59.375535 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:43:10.398345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:43:10.398310 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.376279 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:43:29.376239 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:43:49.944708 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:43:49.944669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:59.375592 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:43:59.375542 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:44:29.376125 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:44:29.376025 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:44:49.990753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:44:49.990715 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.375902 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:44:59.375864 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:45:24.068837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:24.068801 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:29.375559 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:45:29.375522 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:45:31.679362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.679327 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-j7pn2_27d1616c-adc0-4619-93f9-b75ca7d6568b/manager/0.log" Oct 30 19:45:31.898746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.898697 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:45:31.899411 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.899350 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" podUID="27d1616c-adc0-4619-93f9-b75ca7d6568b" containerName="manager" containerID="cri-o://fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae" gracePeriod=10 Oct 30 19:45:31.930782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.930732 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 19:45:31.933966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.933949 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:31.948259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:31.948229 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 19:45:32.049820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.049787 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.049953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.049849 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rg4r\" (UniqueName: \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.146940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.146919 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:45:32.151024 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.151001 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.151113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.151074 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rg4r\" (UniqueName: \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.151361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.151343 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.171842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.171820 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rg4r\" (UniqueName: \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\") pod \"kuadrant-operator-controller-manager-678d87766f-rcvnc\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.252095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.251999 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\") pod \"27d1616c-adc0-4619-93f9-b75ca7d6568b\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " Oct 30 19:45:32.252259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.252109 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\" (UniqueName: \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\") pod \"27d1616c-adc0-4619-93f9-b75ca7d6568b\" (UID: \"27d1616c-adc0-4619-93f9-b75ca7d6568b\") " Oct 30 19:45:32.252410 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.252378 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume" pod "27d1616c-adc0-4619-93f9-b75ca7d6568b" (UID: "27d1616c-adc0-4619-93f9-b75ca7d6568b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:45:32.254206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.254180 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft" pod "27d1616c-adc0-4619-93f9-b75ca7d6568b" (UID: "27d1616c-adc0-4619-93f9-b75ca7d6568b"). InnerVolumeSpecName "kube-api-access-7fvft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:32.286386 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.286362 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:32.353749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.353718 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fvft\" (UniqueName: \"kubernetes.io/projected/27d1616c-adc0-4619-93f9-b75ca7d6568b-kube-api-access-7fvft\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:32.353880 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.353753 5733 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/27d1616c-adc0-4619-93f9-b75ca7d6568b-extensions-socket-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:32.474838 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.474799 5733 generic.go:356] "Generic (PLEG): container finished" podID="27d1616c-adc0-4619-93f9-b75ca7d6568b" containerID="fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae" exitCode=0 Oct 30 19:45:32.475008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.474848 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" event={"ID":"27d1616c-adc0-4619-93f9-b75ca7d6568b","Type":"ContainerDied","Data":"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae"} Oct 30 19:45:32.475008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.474892 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" event={"ID":"27d1616c-adc0-4619-93f9-b75ca7d6568b","Type":"ContainerDied","Data":"5a2cdf74ccc7f9076b62859d08a7e882ac48d1da7837c5eadfb16293221f7750"} Oct 30 19:45:32.475008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.474914 5733 scope.go:122] "RemoveContainer" containerID="fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae" Oct 30 19:45:32.475008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.474848 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2" Oct 30 19:45:32.483781 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.483765 5733 scope.go:122] "RemoveContainer" containerID="fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae" Oct 30 19:45:32.484043 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:45:32.484017 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae\": container with ID starting with fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae not found: ID does not exist" containerID="fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae" Oct 30 19:45:32.484162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.484042 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae"} err="failed to get container status \"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae\": rpc error: code = NotFound desc = could not find container \"fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae\": container with ID starting with fbed4e61fe10b4a81f1b7ac0bc0e0bb94ed61b326593e9d8ff53fd360c427dae not found: ID does not exist" Oct 30 19:45:32.500981 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.500960 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:45:32.509962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.509913 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-j7pn2"] Oct 30 19:45:32.568472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:32.568449 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 19:45:32.570484 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:45:32.570460 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6049f1ce_d07c_4f37_9a3c_dfeff0232cf2.slice/crio-23aff50cfa5c4219ce441dce1483254829a033ba8fab5f22cb61fa99c1a68210 WatchSource:0}: Error finding container 23aff50cfa5c4219ce441dce1483254829a033ba8fab5f22cb61fa99c1a68210: Status 404 returned error can't find the container with id 23aff50cfa5c4219ce441dce1483254829a033ba8fab5f22cb61fa99c1a68210 Oct 30 19:45:33.482749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:33.482715 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerStarted","Data":"ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1"} Oct 30 19:45:33.483155 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:33.482773 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerStarted","Data":"23aff50cfa5c4219ce441dce1483254829a033ba8fab5f22cb61fa99c1a68210"} Oct 30 19:45:33.483155 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:33.482816 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:33.516152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:33.516109 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podStartSLOduration=2.5160965280000003 podStartE2EDuration="2.516096528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:33.51537525 +0000 UTC m=+893.583210818" watchObservedRunningTime="2026-10-30 19:45:33.516096528 +0000 UTC m=+893.583932093" Oct 30 19:45:34.048177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:34.048148 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d1616c-adc0-4619-93f9-b75ca7d6568b" path="/var/lib/kubelet/pods/27d1616c-adc0-4619-93f9-b75ca7d6568b/volumes" Oct 30 19:45:44.489157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:44.489127 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 19:45:50.138334 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:50.138299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.396804 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:50.396724 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.543537 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:50.543501 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.694875 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:50.694787 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.269306 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.269275 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.354834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.354800 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:45:51.355209 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.355194 5733 cpu_manager.go:414] "containerMap: removing container" podUID="27d1616c-adc0-4619-93f9-b75ca7d6568b" containerName="manager" Oct 30 19:45:51.355270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.355210 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d1616c-adc0-4619-93f9-b75ca7d6568b" containerName="manager" Oct 30 19:45:51.355270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.355266 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d1616c-adc0-4619-93f9-b75ca7d6568b" containerName="manager" Oct 30 19:45:51.359734 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.359716 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:45:51.364445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.364420 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:45:51.364602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.364586 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\"" Oct 30 19:45:51.419878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.419850 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.521675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.521606 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbt76\" (UniqueName: \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\") pod \"authorino-56c6c9ddd5-4wzrk\" (UID: \"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:45:51.622806 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.622773 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbt76\" (UniqueName: \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\") pod \"authorino-56c6c9ddd5-4wzrk\" (UID: \"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:45:51.637381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.637345 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbt76\" (UniqueName: \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\") pod \"authorino-56c6c9ddd5-4wzrk\" (UID: \"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc\") " pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:45:51.672178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.672147 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:45:51.946577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.946551 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:45:51.948145 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:45:51.948111 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9a6a7c1_fe82_461b_9600_0444c1dcfcbc.slice/crio-98b74f8fd4f46f65a8bbbcc695e132ea501bcd9374c0af45a3dcfc172d07ec62 WatchSource:0}: Error finding container 98b74f8fd4f46f65a8bbbcc695e132ea501bcd9374c0af45a3dcfc172d07ec62: Status 404 returned error can't find the container with id 98b74f8fd4f46f65a8bbbcc695e132ea501bcd9374c0af45a3dcfc172d07ec62 Oct 30 19:45:51.949517 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:51.949496 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:45:52.575397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:52.575363 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" event={"ID":"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc","Type":"ContainerStarted","Data":"98b74f8fd4f46f65a8bbbcc695e132ea501bcd9374c0af45a3dcfc172d07ec62"} Oct 30 19:45:54.587675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:54.587638 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" event={"ID":"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc","Type":"ContainerStarted","Data":"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182"} Oct 30 19:45:54.608487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:45:54.608437 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" podStartSLOduration=1.872805056 podStartE2EDuration="3.608423804s" totalImagesPullingTime="1.735618748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:54.60744129 +0000 UTC m=+914.675276855" watchObservedRunningTime="2026-10-30 19:45:54.608423804 +0000 UTC m=+914.676259380" Oct 30 19:45:59.375533 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:45:59.375487 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:46:02.473487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:02.473451 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.646508 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:02.646479 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:06.421097 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:06.421048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:06.573887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:06.573847 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:07.316691 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:07.316656 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.497890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:26.497847 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.658010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:26.657975 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.804767 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:26.804693 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:29.375496 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:46:29.375464 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:46:30.058369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:30.058331 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.202005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:30.201972 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.927847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:31.927815 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.073686 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:32.073649 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.125918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:46:50.125878 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:59.376161 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:46:59.376117 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:47:03.667221 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:03.667183 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.823291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:03.823255 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:05.075093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:05.075063 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:05.232881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:05.232843 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:14.862587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:14.862551 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:15.094465 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.094431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:15.225614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.225531 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:15.229560 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.229536 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.239780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.239753 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:47:15.241405 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.241381 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:15.333335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.333299 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zrwq\" (UniqueName: \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.333335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.333343 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.434168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.434134 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.434332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.434239 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zrwq\" (UniqueName: \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.436486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.436462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.450279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.450249 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zrwq\" (UniqueName: \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\") pod \"authorino-6889cbd8c4-bqpx9\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.534329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.533784 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:15.534329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.534242 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:15.809001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.808971 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:15.810606 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:47:15.810581 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5fb317_c322_4bd7_acb9_6ad6d624147d.slice/crio-bc098b470ebf427c6c2f7efe49c84a784d1e384b9018a615b3cba83124f57e45 WatchSource:0}: Error finding container bc098b470ebf427c6c2f7efe49c84a784d1e384b9018a615b3cba83124f57e45: Status 404 returned error can't find the container with id bc098b470ebf427c6c2f7efe49c84a784d1e384b9018a615b3cba83124f57e45 Oct 30 19:47:15.981783 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:15.981747 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" event={"ID":"fc5fb317-c322-4bd7-acb9-6ad6d624147d","Type":"ContainerStarted","Data":"bc098b470ebf427c6c2f7efe49c84a784d1e384b9018a615b3cba83124f57e45"} Oct 30 19:47:16.989409 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:16.989371 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" event={"ID":"fc5fb317-c322-4bd7-acb9-6ad6d624147d","Type":"ContainerStarted","Data":"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf"} Oct 30 19:47:16.989803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:16.989461 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" podUID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" containerName="authorino" containerID="cri-o://8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf" gracePeriod=30 Oct 30 19:47:17.013722 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.013678 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" podStartSLOduration=1.008078293 podStartE2EDuration="2.013665683s" totalImagesPullingTime="1.00558739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:17.012135224 +0000 UTC m=+997.079970790" watchObservedRunningTime="2026-10-30 19:47:17.013665683 +0000 UTC m=+997.081501251" Oct 30 19:47:17.236913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.236890 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:17.353610 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.353571 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\" (UniqueName: \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\") pod \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " Oct 30 19:47:17.353795 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.353782 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\" (UniqueName: \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\") pod \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\" (UID: \"fc5fb317-c322-4bd7-acb9-6ad6d624147d\") " Oct 30 19:47:17.356051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.356012 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq" pod "fc5fb317-c322-4bd7-acb9-6ad6d624147d" (UID: "fc5fb317-c322-4bd7-acb9-6ad6d624147d"). InnerVolumeSpecName "kube-api-access-7zrwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:17.364494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.364404 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert" pod "fc5fb317-c322-4bd7-acb9-6ad6d624147d" (UID: "fc5fb317-c322-4bd7-acb9-6ad6d624147d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:17.408478 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.408436 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:47:17.408691 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.408662 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" podUID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" containerName="authorino" containerID="cri-o://9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182" gracePeriod=30 Oct 30 19:47:17.454335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.454307 5733 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fc5fb317-c322-4bd7-acb9-6ad6d624147d-tls-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.454335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.454333 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zrwq\" (UniqueName: \"kubernetes.io/projected/fc5fb317-c322-4bd7-acb9-6ad6d624147d-kube-api-access-7zrwq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.643398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.643367 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:47:17.757664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.757633 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\" (UniqueName: \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\") pod \"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc\" (UID: \"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc\") " Oct 30 19:47:17.759659 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.759635 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76" pod "b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" (UID: "b9a6a7c1-fe82-461b-9600-0444c1dcfcbc"). InnerVolumeSpecName "kube-api-access-dbt76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:17.858664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.858630 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbt76\" (UniqueName: \"kubernetes.io/projected/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc-kube-api-access-dbt76\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:17.996294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.996180 5733 generic.go:356] "Generic (PLEG): container finished" podID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" containerID="9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182" exitCode=0 Oct 30 19:47:17.996294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.996235 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" Oct 30 19:47:17.996294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.996270 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" event={"ID":"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc","Type":"ContainerDied","Data":"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182"} Oct 30 19:47:17.996814 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.996320 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-4wzrk" event={"ID":"b9a6a7c1-fe82-461b-9600-0444c1dcfcbc","Type":"ContainerDied","Data":"98b74f8fd4f46f65a8bbbcc695e132ea501bcd9374c0af45a3dcfc172d07ec62"} Oct 30 19:47:17.996814 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.996343 5733 scope.go:122] "RemoveContainer" containerID="9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182" Oct 30 19:47:17.997901 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.997874 5733 generic.go:356] "Generic (PLEG): container finished" podID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" containerID="8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf" exitCode=0 Oct 30 19:47:17.998041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.997918 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" Oct 30 19:47:17.998041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.997960 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" event={"ID":"fc5fb317-c322-4bd7-acb9-6ad6d624147d","Type":"ContainerDied","Data":"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf"} Oct 30 19:47:17.998041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:17.998016 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-bqpx9" event={"ID":"fc5fb317-c322-4bd7-acb9-6ad6d624147d","Type":"ContainerDied","Data":"bc098b470ebf427c6c2f7efe49c84a784d1e384b9018a615b3cba83124f57e45"} Oct 30 19:47:18.006695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.006358 5733 scope.go:122] "RemoveContainer" containerID="9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182" Oct 30 19:47:18.006695 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:47:18.006649 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182\": container with ID starting with 9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182 not found: ID does not exist" containerID="9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182" Oct 30 19:47:18.006695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.006681 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182"} err="failed to get container status \"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182\": rpc error: code = NotFound desc = could not find container \"9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182\": container with ID starting with 9951dc2af05dc2560e608020de24a636014187ec36105142600d6d15434b4182 not found: ID does not exist" Oct 30 19:47:18.006861 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.006706 5733 scope.go:122] "RemoveContainer" containerID="8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf" Oct 30 19:47:18.014887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.014866 5733 scope.go:122] "RemoveContainer" containerID="8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf" Oct 30 19:47:18.015188 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:47:18.015165 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf\": container with ID starting with 8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf not found: ID does not exist" containerID="8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf" Oct 30 19:47:18.015272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.015197 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf"} err="failed to get container status \"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf\": rpc error: code = NotFound desc = could not find container \"8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf\": container with ID starting with 8d530c124242742717f5ed91a6926ea75d9b45bbb69a3be7edd9ddee42c744bf not found: ID does not exist" Oct 30 19:47:18.049070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.049028 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:47:18.050717 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.050693 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-4wzrk"] Oct 30 19:47:18.075887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.075862 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:18.079486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:18.079462 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-bqpx9"] Oct 30 19:47:20.053754 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:20.053719 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" path="/var/lib/kubelet/pods/b9a6a7c1-fe82-461b-9600-0444c1dcfcbc/volumes" Oct 30 19:47:20.054149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:20.054041 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" path="/var/lib/kubelet/pods/fc5fb317-c322-4bd7-acb9-6ad6d624147d/volumes" Oct 30 19:47:29.375287 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:47:29.375208 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:47:41.038332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038299 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 19:47:41.038709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038653 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" containerName="authorino" Oct 30 19:47:41.038709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038664 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" containerName="authorino" Oct 30 19:47:41.038709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038686 5733 cpu_manager.go:414] "containerMap: removing container" podUID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" containerName="authorino" Oct 30 19:47:41.038709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038691 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" containerName="authorino" Oct 30 19:47:41.038834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038770 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5fb317-c322-4bd7-acb9-6ad6d624147d" containerName="authorino" Oct 30 19:47:41.038834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.038783 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9a6a7c1-fe82-461b-9600-0444c1dcfcbc" containerName="authorino" Oct 30 19:47:41.043019 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.043003 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.049568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.049545 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:47:41.050039 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050016 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\"" Oct 30 19:47:41.050141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050117 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050171 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050201 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050228 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050358 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050473 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmg7z\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050529 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050562 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.050653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.050605 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.056997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.056973 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 19:47:41.151565 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151525 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.151744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151590 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.151744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151636 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.151744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151658 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.151744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151684 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.151744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151705 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151866 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmg7z\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151932 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.151963 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.152007 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.152200 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.152230 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.152321 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.152726 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.152702 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.154036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.154015 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.154394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.154374 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.167702 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.167674 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.167773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.167679 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmg7z\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\") pod \"maas-default-gateway-openshift-default-57f78bcd55-gdnn8\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.356373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.356344 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:41.641897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:41.641867 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 19:47:41.643563 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:47:41.643540 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7489a135_99a1_41d0_8b09_1288c18ecf8b.slice/crio-1e617e7b8601a82567128068ac4b561ea60a6a3710dccfaa359f9198965ed10f WatchSource:0}: Error finding container 1e617e7b8601a82567128068ac4b561ea60a6a3710dccfaa359f9198965ed10f: Status 404 returned error can't find the container with id 1e617e7b8601a82567128068ac4b561ea60a6a3710dccfaa359f9198965ed10f Oct 30 19:47:42.116658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:42.116623 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" event={"ID":"7489a135-99a1-41d0-8b09-1288c18ecf8b","Type":"ContainerStarted","Data":"1e617e7b8601a82567128068ac4b561ea60a6a3710dccfaa359f9198965ed10f"} Oct 30 19:47:44.665398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:44.665360 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:44.665701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:44.665434 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:44.665701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:44.665472 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:45.139531 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:45.139491 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" event={"ID":"7489a135-99a1-41d0-8b09-1288c18ecf8b","Type":"ContainerStarted","Data":"dab333a75bc732f29a57f9f02819c0a628d87bdec8733309ce2d719cd08572b8"} Oct 30 19:47:45.172357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:45.172297 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podStartSLOduration=1.153184114 podStartE2EDuration="4.172282907s" totalImagesPullingTime="3.019098793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:45.171465801 +0000 UTC m=+1025.239301369" watchObservedRunningTime="2026-10-30 19:47:45.172282907 +0000 UTC m=+1025.240118476" Oct 30 19:47:45.357175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:45.357133 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:45.358542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:45.358514 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:45.358677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:45.358563 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:46.357732 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:46.357700 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:46.358258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:46.357757 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:47.357329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:47.357278 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:47.357554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:47.357375 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:48.357593 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:48.357555 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:48.357980 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:48.357611 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:49.357630 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:49.357594 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:49.358024 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:49.357649 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:50.191851 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:50.191796 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:50.357625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:50.357587 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 30 19:47:50.357800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:50.357656 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 30 19:47:51.361135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:51.361106 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:51.361614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:51.361494 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:51.362134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:51.362116 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 19:47:52.293186 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:52.293157 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.451506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:52.451473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:56.083209 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.083178 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 19:47:56.086985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.086957 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.112336 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.112307 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 19:47:56.147808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.147778 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 19:47:56.151395 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.151379 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.172165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.172140 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 19:47:56.195671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195644 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.195784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195686 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.195784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195769 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.195872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195798 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.195872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195822 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.195872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195859 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.195996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195880 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.195996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195946 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.195996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.195968 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.196163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196031 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.196163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196085 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.196163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196149 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.196284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196205 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.196284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196230 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.196284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196245 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.196284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196270 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.196429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196309 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92f9\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.196429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.196325 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297222 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297260 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297279 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297295 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297310 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297344 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92f9\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297371 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297432 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297469 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297499 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297519 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297543 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297566 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297605 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297633 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297668 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297703 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297713 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297818 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.297993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.297990 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.298490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298130 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.298490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298183 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.298490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298350 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.298490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298424 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.298715 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298525 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.298715 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298536 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.298888 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298749 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.298888 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.298762 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.299895 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.299868 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.300470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.300449 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.300602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.300585 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.300645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.300587 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.311945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.311919 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92f9\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.312548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.312518 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-zj7hb\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.312958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.312936 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.314212 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.314189 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-v484k\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.399443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.399418 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:56.464351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.464323 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:56.685371 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.685342 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 19:47:56.687263 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:47:56.687240 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35a6c5c6_c04a_40b6_a97a_828c0bd2d86c.slice/crio-a38c0bd5c6e4132a3396c3f425d998c0bbd0baf8383eb9a44cf0befb9a9853af WatchSource:0}: Error finding container a38c0bd5c6e4132a3396c3f425d998c0bbd0baf8383eb9a44cf0befb9a9853af: Status 404 returned error can't find the container with id a38c0bd5c6e4132a3396c3f425d998c0bbd0baf8383eb9a44cf0befb9a9853af Oct 30 19:47:56.690219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.690186 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:56.690318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.690246 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:56.690318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.690278 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:56.749845 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.749823 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 19:47:56.752181 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:47:56.752159 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d64cc3_7d77_4972_8df4_4f7040a5de6f.slice/crio-e57cd44d46e81107243d368c9734040f98c567b7b58014054d0e4986b7f79c51 WatchSource:0}: Error finding container e57cd44d46e81107243d368c9734040f98c567b7b58014054d0e4986b7f79c51: Status 404 returned error can't find the container with id e57cd44d46e81107243d368c9734040f98c567b7b58014054d0e4986b7f79c51 Oct 30 19:47:56.756133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.756100 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:56.756214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.756166 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:56.756214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:56.756196 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:47:57.199451 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.199416 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" event={"ID":"99d64cc3-7d77-4972-8df4-4f7040a5de6f","Type":"ContainerStarted","Data":"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e"} Oct 30 19:47:57.199968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.199947 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" event={"ID":"99d64cc3-7d77-4972-8df4-4f7040a5de6f","Type":"ContainerStarted","Data":"e57cd44d46e81107243d368c9734040f98c567b7b58014054d0e4986b7f79c51"} Oct 30 19:47:57.201377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.200931 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" event={"ID":"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c","Type":"ContainerStarted","Data":"6f5d15c9cde2e1e9e132f5ef785511cd6041e47025ae39e2b4f7ea2b4a9790bc"} Oct 30 19:47:57.201377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.200975 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" event={"ID":"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c","Type":"ContainerStarted","Data":"a38c0bd5c6e4132a3396c3f425d998c0bbd0baf8383eb9a44cf0befb9a9853af"} Oct 30 19:47:57.266139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.266077 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podStartSLOduration=1.266046321 podStartE2EDuration="1.266046321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:57.26516264 +0000 UTC m=+1037.332998208" watchObservedRunningTime="2026-10-30 19:47:57.266046321 +0000 UTC m=+1037.333881888" Oct 30 19:47:57.266791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.266762 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podStartSLOduration=1.266753845 podStartE2EDuration="1.266753845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:57.231444808 +0000 UTC m=+1037.299280376" watchObservedRunningTime="2026-10-30 19:47:57.266753845 +0000 UTC m=+1037.334589412" Oct 30 19:47:57.400538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.400498 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:47:57.401864 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.401842 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:47:57.401946 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.401889 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:47:57.465349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.465223 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:47:57.466466 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.466443 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:47:57.466586 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:57.466490 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:47:58.399891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:58.399855 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:47:58.400302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:58.399908 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:47:58.464972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:58.464932 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:47:58.465173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:58.464983 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:47:59.382704 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:47:59.382662 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:47:59.400243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:59.400122 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:47:59.400243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:59.400166 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:47:59.465878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:59.465845 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:47:59.466067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:47:59.465902 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:48:00.400369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:00.400337 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:48:00.400789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:00.400388 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:48:00.465547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:00.465100 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:48:00.465719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:00.465569 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:48:01.400046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:01.400009 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:48:01.400289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:01.400101 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:48:01.464955 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:01.464911 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:48:01.465392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:01.464987 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:48:02.400716 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:02.400677 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Oct 30 19:48:02.400877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:02.400732 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Oct 30 19:48:02.465619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:02.465580 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 30 19:48:02.466021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:02.465637 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 30 19:48:03.404460 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.404431 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:48:03.404710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.404690 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:48:03.405323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.405302 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 19:48:03.469118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.469094 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:48:03.469582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.469416 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:48:03.470426 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:03.470390 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 19:48:09.729514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:09.729480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.760013 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.759879 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8"] Oct 30 19:48:11.765891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.765846 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.776817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.776782 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8"] Oct 30 19:48:11.776961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.776919 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:48:11.776961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.776919 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z86lw\"" Oct 30 19:48:11.777103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.776987 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:48:11.777471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.777451 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:48:11.878720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.878689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.894017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.893983 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.894163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.894021 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952eb089-1681-4eac-94d4-9f7465666913-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.894163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.894077 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87bs9\" (UniqueName: \"kubernetes.io/projected/952eb089-1681-4eac-94d4-9f7465666913-kube-api-access-87bs9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.894248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.894228 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.894291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.894258 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.894291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.894281 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995420 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995456 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952eb089-1681-4eac-94d4-9f7465666913-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995474 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87bs9\" (UniqueName: \"kubernetes.io/projected/952eb089-1681-4eac-94d4-9f7465666913-kube-api-access-87bs9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995547 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995563 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995585 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.995949 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995926 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.996011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995936 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.996011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.995983 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.997936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.997914 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/952eb089-1681-4eac-94d4-9f7465666913-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:11.998236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:11.998216 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/952eb089-1681-4eac-94d4-9f7465666913-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:12.025600 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:12.025543 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87bs9\" (UniqueName: \"kubernetes.io/projected/952eb089-1681-4eac-94d4-9f7465666913-kube-api-access-87bs9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-q9sb8\" (UID: \"952eb089-1681-4eac-94d4-9f7465666913\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:12.081468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:12.081441 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:12.366994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:12.366967 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8"] Oct 30 19:48:12.369160 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:12.369129 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952eb089_1681_4eac_94d4_9f7465666913.slice/crio-ae33191c3a09d2bce0661901317cc42376cce5d3b686200d1c3d1a9dc65e2930 WatchSource:0}: Error finding container ae33191c3a09d2bce0661901317cc42376cce5d3b686200d1c3d1a9dc65e2930: Status 404 returned error can't find the container with id ae33191c3a09d2bce0661901317cc42376cce5d3b686200d1c3d1a9dc65e2930 Oct 30 19:48:13.386080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.382283 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" event={"ID":"952eb089-1681-4eac-94d4-9f7465666913","Type":"ContainerStarted","Data":"ae33191c3a09d2bce0661901317cc42376cce5d3b686200d1c3d1a9dc65e2930"} Oct 30 19:48:13.867218 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.867180 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs"] Oct 30 19:48:13.871492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.871465 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:13.884425 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.884379 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 19:48:13.885615 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.885539 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs"] Oct 30 19:48:13.970539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:13.970507 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.033453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033410 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vcsl\" (UniqueName: \"kubernetes.io/projected/cd58d3a0-2332-494d-85f0-e48f263ac29c-kube-api-access-4vcsl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.033634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033514 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.033634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033567 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.033634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033591 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.033777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033650 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.033777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.033742 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd58d3a0-2332-494d-85f0-e48f263ac29c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.134761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134696 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.134761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134749 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.134958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134780 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.134958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134829 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.134958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134924 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd58d3a0-2332-494d-85f0-e48f263ac29c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.135099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.134968 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vcsl\" (UniqueName: \"kubernetes.io/projected/cd58d3a0-2332-494d-85f0-e48f263ac29c-kube-api-access-4vcsl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.135168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.135148 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.135233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.135195 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.135293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.135281 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.136976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.136953 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd58d3a0-2332-494d-85f0-e48f263ac29c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.137360 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.137340 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd58d3a0-2332-494d-85f0-e48f263ac29c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.148596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.148569 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vcsl\" (UniqueName: \"kubernetes.io/projected/cd58d3a0-2332-494d-85f0-e48f263ac29c-kube-api-access-4vcsl\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-nrlrs\" (UID: \"cd58d3a0-2332-494d-85f0-e48f263ac29c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.203688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.203667 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:14.389102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.389007 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" event={"ID":"952eb089-1681-4eac-94d4-9f7465666913","Type":"ContainerStarted","Data":"f17b6c77bc29a31d9bf6003c1189221fe0a5b1c22720a2b572ab40fcbd0956b7"} Oct 30 19:48:14.412470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.412424 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podStartSLOduration=1.5964543450000002 podStartE2EDuration="3.412410957s" totalImagesPullingTime="1.815956612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:14.4110761 +0000 UTC m=+1054.478911666" watchObservedRunningTime="2026-10-30 19:48:14.412410957 +0000 UTC m=+1054.480246525" Oct 30 19:48:14.487292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.487262 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs"] Oct 30 19:48:14.489129 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:14.489104 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd58d3a0_2332_494d_85f0_e48f263ac29c.slice/crio-cc3507d0c58e5884f276918083f08082f7f9c419d4ede71f3087c97037969ca9 WatchSource:0}: Error finding container cc3507d0c58e5884f276918083f08082f7f9c419d4ede71f3087c97037969ca9: Status 404 returned error can't find the container with id cc3507d0c58e5884f276918083f08082f7f9c419d4ede71f3087c97037969ca9 Oct 30 19:48:14.732741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.732671 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875"] Oct 30 19:48:14.736325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.736298 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.748641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.748569 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 19:48:14.751410 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.751381 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875"] Oct 30 19:48:14.842133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842105 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.842227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842158 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.842272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842232 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.842307 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842269 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cpxh\" (UniqueName: \"kubernetes.io/projected/294d869b-a6be-48ff-95f3-8123552c5d4f-kube-api-access-2cpxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.842428 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842413 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.842465 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.842438 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294d869b-a6be-48ff-95f3-8123552c5d4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.892464 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.892421 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.943712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943676 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.943712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943714 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294d869b-a6be-48ff-95f3-8123552c5d4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.943921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943740 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.943921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943790 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.943921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943893 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.944031 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.943928 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cpxh\" (UniqueName: \"kubernetes.io/projected/294d869b-a6be-48ff-95f3-8123552c5d4f-kube-api-access-2cpxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.944136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.944119 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.944175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.944152 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.944269 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.944234 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.946205 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.946187 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/294d869b-a6be-48ff-95f3-8123552c5d4f-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.946267 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.946240 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/294d869b-a6be-48ff-95f3-8123552c5d4f-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:14.958285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:14.958259 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cpxh\" (UniqueName: \"kubernetes.io/projected/294d869b-a6be-48ff-95f3-8123552c5d4f-kube-api-access-2cpxh\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-tn875\" (UID: \"294d869b-a6be-48ff-95f3-8123552c5d4f\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:15.054536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.054458 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:15.082520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.082483 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:15.384131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.384051 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875"] Oct 30 19:48:15.386739 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:15.386708 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294d869b_a6be_48ff_95f3_8123552c5d4f.slice/crio-a41213db175d777d4e9fd626d468d7e651d33654114fd74056b3854bdd155b21 WatchSource:0}: Error finding container a41213db175d777d4e9fd626d468d7e651d33654114fd74056b3854bdd155b21: Status 404 returned error can't find the container with id a41213db175d777d4e9fd626d468d7e651d33654114fd74056b3854bdd155b21 Oct 30 19:48:15.400274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.400248 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" event={"ID":"294d869b-a6be-48ff-95f3-8123552c5d4f","Type":"ContainerStarted","Data":"a41213db175d777d4e9fd626d468d7e651d33654114fd74056b3854bdd155b21"} Oct 30 19:48:15.402782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.402749 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" event={"ID":"cd58d3a0-2332-494d-85f0-e48f263ac29c","Type":"ContainerStarted","Data":"9596242cb4869082c0c9f67ce4ceea61295c8ddabe247e8bbfd1022a71e6421c"} Oct 30 19:48:15.402883 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.402806 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" event={"ID":"cd58d3a0-2332-494d-85f0-e48f263ac29c","Type":"ContainerStarted","Data":"cc3507d0c58e5884f276918083f08082f7f9c419d4ede71f3087c97037969ca9"} Oct 30 19:48:15.436132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:15.436048 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podStartSLOduration=2.140976331 podStartE2EDuration="2.436030002s" totalImagesPullingTime="295.053671ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:15.433043419 +0000 UTC m=+1055.500878990" watchObservedRunningTime="2026-10-30 19:48:15.436030002 +0000 UTC m=+1055.503865659" Oct 30 19:48:16.204022 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:16.203991 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:16.412440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:16.411877 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" event={"ID":"294d869b-a6be-48ff-95f3-8123552c5d4f","Type":"ContainerStarted","Data":"5f52993c8e907934e31ae4dbab1ab42b634f41d606bdce63302566b1d98bafe9"} Oct 30 19:48:16.442541 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:16.442486 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podStartSLOduration=2.078986529 podStartE2EDuration="2.44247184s" totalImagesPullingTime="363.485311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:16.438714756 +0000 UTC m=+1056.506550325" watchObservedRunningTime="2026-10-30 19:48:16.44247184 +0000 UTC m=+1056.510307407" Oct 30 19:48:16.527670 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:16.527588 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:17.054783 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:17.054749 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:17.950110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:17.950074 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:19.850662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.850624 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq"] Oct 30 19:48:19.861670 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.861637 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.866692 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.866664 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq"] Oct 30 19:48:19.869790 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.869768 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.973832 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.973896 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.973957 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.973997 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.974039 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.974150 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkkw\" (UniqueName: \"kubernetes.io/projected/0c7a23be-1c57-4b4f-a494-0ff740d58fca-kube-api-access-vwkkw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:19.983490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:19.974598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087336 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087402 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087445 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087487 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkkw\" (UniqueName: \"kubernetes.io/projected/0c7a23be-1c57-4b4f-a494-0ff740d58fca-kube-api-access-vwkkw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087560 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.087601 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.088041 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.088317 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.090251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.088540 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.096332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.095645 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c7a23be-1c57-4b4f-a494-0ff740d58fca-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.096332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.096254 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c7a23be-1c57-4b4f-a494-0ff740d58fca-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.103738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.103677 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkkw\" (UniqueName: \"kubernetes.io/projected/0c7a23be-1c57-4b4f-a494-0ff740d58fca-kube-api-access-vwkkw\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq\" (UID: \"0c7a23be-1c57-4b4f-a494-0ff740d58fca\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.184468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.184293 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:20.794131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:20.793496 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq"] Oct 30 19:48:21.388285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:21.388247 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.502870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:21.502773 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" event={"ID":"0c7a23be-1c57-4b4f-a494-0ff740d58fca","Type":"ContainerStarted","Data":"494c4b1dfb0b16bda9ef36dfab06b8760f60eb791fcda1191f792d12a69789da"} Oct 30 19:48:21.502870 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:21.502844 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" event={"ID":"0c7a23be-1c57-4b4f-a494-0ff740d58fca","Type":"ContainerStarted","Data":"ef5d0682db6a154139c659180633014de544510b19570b75fe830e3480f46155"} Oct 30 19:48:21.530159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:21.530103 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podStartSLOduration=2.161530371 podStartE2EDuration="2.530087704s" totalImagesPullingTime="368.557333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:21.529016523 +0000 UTC m=+1061.596852082" watchObservedRunningTime="2026-10-30 19:48:21.530087704 +0000 UTC m=+1061.597923271" Oct 30 19:48:22.082129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:22.082090 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:22.095515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:22.095491 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:22.185437 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:22.185406 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:22.519244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:22.519215 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 19:48:24.204428 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:24.204394 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:24.245080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:24.244391 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:24.584136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:24.583503 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 19:48:25.059081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:25.058489 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:25.091492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:25.091462 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:25.554379 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:25.554348 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 19:48:29.375847 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:48:29.375815 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:48:30.184858 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:30.184828 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:30.197382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:30.197358 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:30.580237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:30.580157 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 19:48:35.240975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.240938 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fc95fd96f-2wntz"] Oct 30 19:48:35.244806 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.244781 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.252370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.252339 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc95fd96f-2wntz"] Oct 30 19:48:35.252948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.252883 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 19:48:35.253021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.252959 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\"" Oct 30 19:48:35.272574 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.272540 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4cm2\" (UniqueName: \"kubernetes.io/projected/0125b6d4-4e87-4bba-b004-48ad867d787d-kube-api-access-d4cm2\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.272722 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.272599 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0125b6d4-4e87-4bba-b004-48ad867d787d-tls-cert\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.374234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.374196 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0125b6d4-4e87-4bba-b004-48ad867d787d-tls-cert\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.374404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.374318 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4cm2\" (UniqueName: \"kubernetes.io/projected/0125b6d4-4e87-4bba-b004-48ad867d787d-kube-api-access-d4cm2\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.376787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.376760 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0125b6d4-4e87-4bba-b004-48ad867d787d-tls-cert\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.388323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.388300 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4cm2\" (UniqueName: \"kubernetes.io/projected/0125b6d4-4e87-4bba-b004-48ad867d787d-kube-api-access-d4cm2\") pod \"authorino-5fc95fd96f-2wntz\" (UID: \"0125b6d4-4e87-4bba-b004-48ad867d787d\") " pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.559544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.559450 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fc95fd96f-2wntz" Oct 30 19:48:35.836515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:35.836480 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fc95fd96f-2wntz"] Oct 30 19:48:35.838305 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:35.838274 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0125b6d4_4e87_4bba_b004_48ad867d787d.slice/crio-beb73cd6a95a6b7622189902050ad0fbf7f9195bfc6080ac6fd831b38e4e1678 WatchSource:0}: Error finding container beb73cd6a95a6b7622189902050ad0fbf7f9195bfc6080ac6fd831b38e4e1678: Status 404 returned error can't find the container with id beb73cd6a95a6b7622189902050ad0fbf7f9195bfc6080ac6fd831b38e4e1678 Oct 30 19:48:36.601595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:36.601553 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc95fd96f-2wntz" event={"ID":"0125b6d4-4e87-4bba-b004-48ad867d787d","Type":"ContainerStarted","Data":"beb73cd6a95a6b7622189902050ad0fbf7f9195bfc6080ac6fd831b38e4e1678"} Oct 30 19:48:37.608749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:37.608707 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fc95fd96f-2wntz" event={"ID":"0125b6d4-4e87-4bba-b004-48ad867d787d","Type":"ContainerStarted","Data":"3a53584d42be8291a473cbed2c57545180df2b69cdfcc05b2f42e9006c424f46"} Oct 30 19:48:37.649405 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:37.649335 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fc95fd96f-2wntz" podStartSLOduration=1.7939473590000001 podStartE2EDuration="2.649311059s" totalImagesPullingTime="855.3637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:37.646570656 +0000 UTC m=+1077.714406237" watchObservedRunningTime="2026-10-30 19:48:37.649311059 +0000 UTC m=+1077.717146628" Oct 30 19:48:38.411358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.411318 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:48:38.415686 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.415650 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447286 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447460 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447703 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447744 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447817 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-f4tgk\"" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.447925 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 19:48:38.452081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.448986 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.506300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.506266 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.506467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.506327 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlz9j\" (UniqueName: \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.607501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.607461 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.607684 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.607568 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlz9j\" (UniqueName: \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.610175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.610149 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.626171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.626147 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlz9j\" (UniqueName: \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\") pod \"ai-gateway-controller-66dd76bf94-6x5pf\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:38.636397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.636362 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.757769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:38.757684 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:39.073506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:39.073431 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:48:39.077294 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:39.077260 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d43eb33_7440_4d25_94c2_b95a7e01684e.slice/crio-760e74a23afb92ca535e1eeaa1e8cf3f0e739695bc259dd8df7a06f6e6771453 WatchSource:0}: Error finding container 760e74a23afb92ca535e1eeaa1e8cf3f0e739695bc259dd8df7a06f6e6771453: Status 404 returned error can't find the container with id 760e74a23afb92ca535e1eeaa1e8cf3f0e739695bc259dd8df7a06f6e6771453 Oct 30 19:48:39.622579 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:39.622537 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" event={"ID":"0d43eb33-7440-4d25-94c2-b95a7e01684e","Type":"ContainerStarted","Data":"760e74a23afb92ca535e1eeaa1e8cf3f0e739695bc259dd8df7a06f6e6771453"} Oct 30 19:48:43.287266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:43.287243 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 19:48:43.650376 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:43.650335 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" event={"ID":"0d43eb33-7440-4d25-94c2-b95a7e01684e","Type":"ContainerStarted","Data":"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589"} Oct 30 19:48:43.650569 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:43.650427 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:43.674233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:43.674182 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" podStartSLOduration=1.470603294 podStartE2EDuration="5.674169754s" totalImagesPullingTime="4.20356646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:43.671451684 +0000 UTC m=+1083.739287253" watchObservedRunningTime="2026-10-30 19:48:43.674169754 +0000 UTC m=+1083.742005322" Oct 30 19:48:45.215865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.215814 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:45.320940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.320906 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6slx7"] Oct 30 19:48:45.324946 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.324927 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.337291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.337265 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6slx7"] Oct 30 19:48:45.378401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.378364 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 19:48:45.378566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.378441 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4qwx\"" Oct 30 19:48:45.384576 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.384544 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ce4b26-7e58-442e-85bc-21bc1535c2fd-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.384742 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.384607 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgzw\" (UniqueName: \"kubernetes.io/projected/23ce4b26-7e58-442e-85bc-21bc1535c2fd-kube-api-access-bfgzw\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.471647 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.471556 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:45.485953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.485924 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgzw\" (UniqueName: \"kubernetes.io/projected/23ce4b26-7e58-442e-85bc-21bc1535c2fd-kube-api-access-bfgzw\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.486111 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.486080 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ce4b26-7e58-442e-85bc-21bc1535c2fd-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.486761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.486737 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ce4b26-7e58-442e-85bc-21bc1535c2fd-plugins-config-volume\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.502226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.502204 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgzw\" (UniqueName: \"kubernetes.io/projected/23ce4b26-7e58-442e-85bc-21bc1535c2fd-kube-api-access-bfgzw\") pod \"payload-processing-5f78687d5b-6slx7\" (UID: \"23ce4b26-7e58-442e-85bc-21bc1535c2fd\") " pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.638116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.638046 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:45.920092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:45.920046 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-6slx7"] Oct 30 19:48:45.922019 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:48:45.921986 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ce4b26_7e58_442e_85bc_21bc1535c2fd.slice/crio-85b2a744363b238756d01bb6cb2c2805e6357fc36eb861c45548dbae0706d0ea WatchSource:0}: Error finding container 85b2a744363b238756d01bb6cb2c2805e6357fc36eb861c45548dbae0706d0ea: Status 404 returned error can't find the container with id 85b2a744363b238756d01bb6cb2c2805e6357fc36eb861c45548dbae0706d0ea Oct 30 19:48:46.672599 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:46.672561 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" event={"ID":"23ce4b26-7e58-442e-85bc-21bc1535c2fd","Type":"ContainerStarted","Data":"85b2a744363b238756d01bb6cb2c2805e6357fc36eb861c45548dbae0706d0ea"} Oct 30 19:48:48.687883 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:48.687846 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" event={"ID":"23ce4b26-7e58-442e-85bc-21bc1535c2fd","Type":"ContainerStarted","Data":"67189c6bd9664359c2dd2b54d80d6404cc5572cc67ed43251d5110a2a18baa8a"} Oct 30 19:48:48.688306 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:48.687978 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:48:48.710739 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:48.710687 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" podStartSLOduration=1.632108238 podStartE2EDuration="3.710674113s" totalImagesPullingTime="2.078565875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:48.70965251 +0000 UTC m=+1088.777488082" watchObservedRunningTime="2026-10-30 19:48:48.710674113 +0000 UTC m=+1088.778509680" Oct 30 19:48:50.238393 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:50.238361 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:54.659578 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:54.659500 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:48:59.375952 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:48:59.375917 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:48:59.697828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:48:59.697750 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 19:49:09.105298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.105259 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:49:09.105872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.105498 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" podUID="0d43eb33-7440-4d25-94c2-b95a7e01684e" containerName="manager" containerID="cri-o://d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589" gracePeriod=30 Oct 30 19:49:09.348271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.348244 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:49:09.521096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.520986 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\" (UniqueName: \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\") pod \"0d43eb33-7440-4d25-94c2-b95a7e01684e\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " Oct 30 19:49:09.521280 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.521094 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\") pod \"0d43eb33-7440-4d25-94c2-b95a7e01684e\" (UID: \"0d43eb33-7440-4d25-94c2-b95a7e01684e\") " Oct 30 19:49:09.523289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.523259 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j" pod "0d43eb33-7440-4d25-94c2-b95a7e01684e" (UID: "0d43eb33-7440-4d25-94c2-b95a7e01684e"). InnerVolumeSpecName "kube-api-access-wlz9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:09.523289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.523265 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs" pod "0d43eb33-7440-4d25-94c2-b95a7e01684e" (UID: "0d43eb33-7440-4d25-94c2-b95a7e01684e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:09.621975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.621945 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d43eb33-7440-4d25-94c2-b95a7e01684e-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:09.621975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.621971 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlz9j\" (UniqueName: \"kubernetes.io/projected/0d43eb33-7440-4d25-94c2-b95a7e01684e-kube-api-access-wlz9j\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:09.800749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.800661 5733 generic.go:356] "Generic (PLEG): container finished" podID="0d43eb33-7440-4d25-94c2-b95a7e01684e" containerID="d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589" exitCode=0 Oct 30 19:49:09.800749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.800708 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" Oct 30 19:49:09.800953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.800754 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" event={"ID":"0d43eb33-7440-4d25-94c2-b95a7e01684e","Type":"ContainerDied","Data":"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589"} Oct 30 19:49:09.800953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.800804 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-66dd76bf94-6x5pf" event={"ID":"0d43eb33-7440-4d25-94c2-b95a7e01684e","Type":"ContainerDied","Data":"760e74a23afb92ca535e1eeaa1e8cf3f0e739695bc259dd8df7a06f6e6771453"} Oct 30 19:49:09.800953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.800836 5733 scope.go:122] "RemoveContainer" containerID="d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589" Oct 30 19:49:09.810251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.810231 5733 scope.go:122] "RemoveContainer" containerID="d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589" Oct 30 19:49:09.810502 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:49:09.810480 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589\": container with ID starting with d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589 not found: ID does not exist" containerID="d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589" Oct 30 19:49:09.810571 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.810513 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589"} err="failed to get container status \"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589\": rpc error: code = NotFound desc = could not find container \"d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589\": container with ID starting with d61a529a760124a734a4877107d8a2d4cd53e1e9a0a07a13d53576129da77589 not found: ID does not exist" Oct 30 19:49:09.826455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.826433 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:49:09.830336 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:09.830316 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-66dd76bf94-6x5pf"] Oct 30 19:49:10.048085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:10.048040 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d43eb33-7440-4d25-94c2-b95a7e01684e" path="/var/lib/kubelet/pods/0d43eb33-7440-4d25-94c2-b95a7e01684e/volumes" Oct 30 19:49:10.896641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:10.896587 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.471536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:11.471488 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.113640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:13.113604 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.319832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:13.319799 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:14.881919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:14.881883 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.070204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:15.070144 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.812838 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:17.812787 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:29.375941 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:49:29.375903 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:49:38.975801 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:38.975759 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.146570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.146529 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.685983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.685930 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.762698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.762665 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 19:49:39.763269 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.763253 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0d43eb33-7440-4d25-94c2-b95a7e01684e" containerName="manager" Oct 30 19:49:39.763325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.763271 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d43eb33-7440-4d25-94c2-b95a7e01684e" containerName="manager" Oct 30 19:49:39.763367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.763358 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d43eb33-7440-4d25-94c2-b95a7e01684e" containerName="manager" Oct 30 19:49:39.766676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.766654 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.777380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.777359 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\"" Oct 30 19:49:39.786944 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.786922 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 19:49:39.913313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913261 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv6bv\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913316 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913424 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913500 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913538 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913577 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913787 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913822 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.913904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.913880 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:39.955182 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:39.955098 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.015201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015164 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015219 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv6bv\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015251 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015274 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015334 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015388 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015530 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015628 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.015677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015652 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.016131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.015893 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.016131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.016030 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.016251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.016127 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.016251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.016150 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.016251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.016150 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.017787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.017762 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.017905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.017768 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.031412 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.031386 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.031572 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.031551 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv6bv\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.084030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.084004 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\"" Oct 30 19:49:40.090237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.090200 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:40.107528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.107505 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 19:49:40.127844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.127817 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.143400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.143357 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 19:49:40.170665 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.169762 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a1d885-openshift-default-dockercfg-stn6c\"" Oct 30 19:49:40.171393 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.171361 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.217549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217513 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217631 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217680 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217770 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217822 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217865 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv8n\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.217978 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.218022 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.218580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.218196 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319342 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319401 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319428 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319454 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319583 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319641 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319686 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv8n\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319734 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319774 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319785 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.319973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319907 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.320168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.319997 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.320168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.320132 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.320275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.320242 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.321968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.321945 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.322105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.322038 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.339629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.339599 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv8n\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.340212 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.340191 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.378861 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.378825 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.405979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.405950 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 19:49:40.407960 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:40.407929 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d233c6_d1fc_4b59_8998_afe61144f065.slice/crio-2e1a8ef944300868f3eefcb491d2a4c9aaffa01dd37bb96820aced91596d3b79 WatchSource:0}: Error finding container 2e1a8ef944300868f3eefcb491d2a4c9aaffa01dd37bb96820aced91596d3b79: Status 404 returned error can't find the container with id 2e1a8ef944300868f3eefcb491d2a4c9aaffa01dd37bb96820aced91596d3b79 Oct 30 19:49:40.410370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.410334 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.410445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.410416 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.410481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.410455 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.459809 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.459785 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:40.512177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.510503 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 19:49:40.515292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.515220 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.530897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.528393 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f933e-openshift-default-dockercfg-dspvq\"" Oct 30 19:49:40.536399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.536374 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 19:49:40.622557 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622522 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622755 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622674 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622752 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622776 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622791 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7sv2\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622895 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622916 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.622966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.622937 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.623140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.623075 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.638458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.638423 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.724232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724159 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724193 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724251 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724276 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724296 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724311 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7sv2\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724515 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724577 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724630 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724716 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724820 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.724842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.724831 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.725133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.725107 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.725326 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.725303 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.726898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.726868 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.726991 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.726975 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.754808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.754780 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7sv2\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.755330 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.755305 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.801895 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.801855 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 19:49:40.820383 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:40.820349 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90effab3_a893_40cd_b620_673575f1d679.slice/crio-77f017ec4cc1a98ddc94f57bc7660fdde23e7f8bf9b4355fa0fd544aea65dd8a WatchSource:0}: Error finding container 77f017ec4cc1a98ddc94f57bc7660fdde23e7f8bf9b4355fa0fd544aea65dd8a: Status 404 returned error can't find the container with id 77f017ec4cc1a98ddc94f57bc7660fdde23e7f8bf9b4355fa0fd544aea65dd8a Oct 30 19:49:40.822045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.822011 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.822207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.822109 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.822207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.822159 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:40.835491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.835461 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:40.971694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.971655 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerStarted","Data":"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd"} Oct 30 19:49:40.971856 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.971711 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerStarted","Data":"2e1a8ef944300868f3eefcb491d2a4c9aaffa01dd37bb96820aced91596d3b79"} Oct 30 19:49:40.974024 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.973989 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerStarted","Data":"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f"} Oct 30 19:49:40.974176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:40.974043 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerStarted","Data":"77f017ec4cc1a98ddc94f57bc7660fdde23e7f8bf9b4355fa0fd544aea65dd8a"} Oct 30 19:49:41.014244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.014143 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podStartSLOduration=2.014129024 podStartE2EDuration="2.014129024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:41.01300625 +0000 UTC m=+1141.080841820" watchObservedRunningTime="2026-10-30 19:49:41.014129024 +0000 UTC m=+1141.081964591" Oct 30 19:49:41.052847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.052819 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:41.066975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.066920 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podStartSLOduration=1.066907584 podStartE2EDuration="1.066907584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:41.065228376 +0000 UTC m=+1141.133063947" watchObservedRunningTime="2026-10-30 19:49:41.066907584 +0000 UTC m=+1141.134743152" Oct 30 19:49:41.090479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.090448 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:41.095714 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.095689 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:41.183922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.183895 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 19:49:41.195393 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:41.193989 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f49df1b_29d9_48cc_b270_00a78df3439e.slice/crio-caf43166f3c3ae8de65273cab246653177b029cc696ac7297f17fb5ff8a17ecb WatchSource:0}: Error finding container caf43166f3c3ae8de65273cab246653177b029cc696ac7297f17fb5ff8a17ecb: Status 404 returned error can't find the container with id caf43166f3c3ae8de65273cab246653177b029cc696ac7297f17fb5ff8a17ecb Oct 30 19:49:41.197407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.197366 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:41.197525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.197454 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:41.197525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.197495 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:49:41.380934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.380901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:41.460286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.460251 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:41.465768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.465743 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:41.981917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.981880 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerStarted","Data":"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1"} Oct 30 19:49:41.982098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.981952 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerStarted","Data":"caf43166f3c3ae8de65273cab246653177b029cc696ac7297f17fb5ff8a17ecb"} Oct 30 19:49:41.982517 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.982493 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:41.982712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.982685 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:41.983268 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.983251 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:49:41.983364 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:41.983346 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:49:42.021474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:42.021430 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podStartSLOduration=2.021417553 podStartE2EDuration="2.021417553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:42.020729652 +0000 UTC m=+1142.088565220" watchObservedRunningTime="2026-10-30 19:49:42.021417553 +0000 UTC m=+1142.089253174" Oct 30 19:49:42.836443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:42.836402 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:42.843199 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:42.843169 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:42.994365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:42.994336 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:42.995180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:42.995159 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:49:43.087807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:43.087672 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.322849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:43.322816 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.688459 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:43.688427 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.936974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:43.936937 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.178643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.178607 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.368329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.368290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.672320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.672288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.731252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.731219 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 19:49:44.734845 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.734829 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:44.747023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.746994 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d5e3e5\"" Oct 30 19:49:44.752874 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.752850 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 19:49:44.840520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.840486 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 19:49:44.844030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.844012 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:44.858176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.858150 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 19:49:44.863174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.863146 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d5e3e5-dockercfg-tbc7b\"" Oct 30 19:49:44.880733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.880708 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:44.880867 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.880745 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6l4\" (UniqueName: \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:44.909099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.909052 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.981634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.981539 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrd55\" (UniqueName: \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:44.982112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.981632 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:44.982112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.981677 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:44.982112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.981708 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6l4\" (UniqueName: \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:44.982406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:44.982378 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:45.011803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.011770 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6l4\" (UniqueName: \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\") pod \"payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:45.046938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.046908 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:45.082476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.082447 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrd55\" (UniqueName: \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:45.082636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.082573 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:45.083721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.083694 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:45.098321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.098294 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrd55\" (UniqueName: \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\") pod \"payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:45.154099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.154052 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.158204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.157877 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:45.340743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.340643 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.349090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.344150 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 19:49:45.349090 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:45.345497 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4c6696_0306_4c89_9f3d_81fbdfa0ecfa.slice/crio-a07dac7575211bac9bbda063789fddfca1c5969a4a5123114615821a21a89a4b WatchSource:0}: Error finding container a07dac7575211bac9bbda063789fddfca1c5969a4a5123114615821a21a89a4b: Status 404 returned error can't find the container with id a07dac7575211bac9bbda063789fddfca1c5969a4a5123114615821a21a89a4b Oct 30 19:49:45.457807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.457776 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 19:49:45.459125 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:45.459098 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe5eb2e_3136_4690_815f_ead77c1a5384.slice/crio-efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691 WatchSource:0}: Error finding container efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691: Status 404 returned error can't find the container with id efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691 Oct 30 19:49:45.527781 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.527758 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 19:49:45.531699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.531676 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.546600 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.546576 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3f57c5\"" Oct 30 19:49:45.548113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.548085 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3f57c5-dockercfg-m7n92\"" Oct 30 19:49:45.550779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.550754 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.552480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.552439 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 19:49:45.555263 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.555090 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 19:49:45.564200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.562878 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.575105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.575078 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 19:49:45.691049 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.691014 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vt6s\" (UniqueName: \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.691257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.691099 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9l8\" (UniqueName: \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.691257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.691141 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.691257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.691203 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.738174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.738140 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.792190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.792161 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9l8\" (UniqueName: \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.792325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.792212 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.792325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.792266 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.792432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.792335 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vt6s\" (UniqueName: \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.793093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.793074 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.793162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.793139 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.808453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.808424 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vt6s\" (UniqueName: \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\") pod \"payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.810710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.810688 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9l8\" (UniqueName: \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\") pod \"payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:45.847766 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.847742 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:45.888683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:45.888650 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:46.014656 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.014615 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" event={"ID":"1fe5eb2e-3136-4690-815f-ead77c1a5384","Type":"ContainerStarted","Data":"7811792985c4220199a6babde04642377e65c376b6378e9c11f93f518cffa9ab"} Oct 30 19:49:46.015080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.014669 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" event={"ID":"1fe5eb2e-3136-4690-815f-ead77c1a5384","Type":"ContainerStarted","Data":"efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691"} Oct 30 19:49:46.015553 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.015534 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:46.015682 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.015668 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.017605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.017581 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" event={"ID":"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa","Type":"ContainerStarted","Data":"1edfb42b38ed41db97908a3580b4546b999ed0c797f4ab555f90a0f9861ebf85"} Oct 30 19:49:46.017743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.017627 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" event={"ID":"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa","Type":"ContainerStarted","Data":"a07dac7575211bac9bbda063789fddfca1c5969a4a5123114615821a21a89a4b"} Oct 30 19:49:46.017743 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.017657 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:46.047046 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.046999 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podStartSLOduration=2.046987279 podStartE2EDuration="2.046987279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:46.046558999 +0000 UTC m=+1146.114394569" watchObservedRunningTime="2026-10-30 19:49:46.046987279 +0000 UTC m=+1146.114822848" Oct 30 19:49:46.103424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.103390 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 19:49:46.108286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.108242 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podStartSLOduration=2.108228668 podStartE2EDuration="2.108228668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:46.106905137 +0000 UTC m=+1146.174740716" watchObservedRunningTime="2026-10-30 19:49:46.108228668 +0000 UTC m=+1146.176064235" Oct 30 19:49:46.108945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.108922 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.131002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.130976 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a1d885\"" Oct 30 19:49:46.134802 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.134774 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 19:49:46.173750 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:46.173720 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb700fa9e_7b5a_42f7_9d54_551c147eccc8.slice/crio-9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7 WatchSource:0}: Error finding container 9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7: Status 404 returned error can't find the container with id 9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7 Oct 30 19:49:46.174740 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.174709 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 19:49:46.196239 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:46.196201 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd91d8c7_be08_4975_b46b_70f110ca1bfc.slice/crio-4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f WatchSource:0}: Error finding container 4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f: Status 404 returned error can't find the container with id 4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f Oct 30 19:49:46.196338 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.196277 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.196338 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.196284 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 19:49:46.196469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.196442 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbmn\" (UniqueName: \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.300047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.300007 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbmn\" (UniqueName: \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.300330 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.300172 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.301110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.301039 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.318695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.318661 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.323986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.323953 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbmn\" (UniqueName: \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\") pod \"payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.423830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.423796 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:46.729018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.728993 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 19:49:46.730008 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:46.729981 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7733aeff_8a16_47ab_ae96_b3ee56b1ef38.slice/crio-27ad818d1a5b6dca4ef29b9eb641d728aac405699af455ce07b211ec7370b660 WatchSource:0}: Error finding container 27ad818d1a5b6dca4ef29b9eb641d728aac405699af455ce07b211ec7370b660: Status 404 returned error can't find the container with id 27ad818d1a5b6dca4ef29b9eb641d728aac405699af455ce07b211ec7370b660 Oct 30 19:49:46.775538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.775507 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.892878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.892843 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 19:49:46.896815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.896789 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:46.921547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.921515 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 19:49:46.949732 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.949697 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:46.959392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.959349 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ced832\"" Oct 30 19:49:46.962815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:46.962788 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ced832-dockercfg-kxkvg\"" Oct 30 19:49:47.006530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.006445 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.006530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.006485 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmssh\" (UniqueName: \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.027023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.026985 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" event={"ID":"7733aeff-8a16-47ab-ae96-b3ee56b1ef38","Type":"ContainerStarted","Data":"739bdec92ca443c7e6f44c89b76d3dd4ddefeac50228c0adbb0aef7f3270785f"} Oct 30 19:49:47.027477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.027106 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" event={"ID":"7733aeff-8a16-47ab-ae96-b3ee56b1ef38","Type":"ContainerStarted","Data":"27ad818d1a5b6dca4ef29b9eb641d728aac405699af455ce07b211ec7370b660"} Oct 30 19:49:47.027477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.027141 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:47.029008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.028978 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" event={"ID":"b700fa9e-7b5a-42f7-9d54-551c147eccc8","Type":"ContainerStarted","Data":"45fd828105b297d51743901958169d9c03629a4daec77821757ed259fa55e6e0"} Oct 30 19:49:47.029149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.029029 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" event={"ID":"b700fa9e-7b5a-42f7-9d54-551c147eccc8","Type":"ContainerStarted","Data":"9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7"} Oct 30 19:49:47.029210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.029183 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:47.030704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.030677 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" event={"ID":"bd91d8c7-be08-4975-b46b-70f110ca1bfc","Type":"ContainerStarted","Data":"e9789732e4398411d3ef1eb358945c345562b92e11eb20b3d22ca454e1fa91ff"} Oct 30 19:49:47.030797 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.030726 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" event={"ID":"bd91d8c7-be08-4975-b46b-70f110ca1bfc","Type":"ContainerStarted","Data":"4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f"} Oct 30 19:49:47.030909 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.030812 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:47.067102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.067027 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podStartSLOduration=1.067008767 podStartE2EDuration="1.067008767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:47.063601412 +0000 UTC m=+1147.131436982" watchObservedRunningTime="2026-10-30 19:49:47.067008767 +0000 UTC m=+1147.134844338" Oct 30 19:49:47.108827 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.108769 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podStartSLOduration=2.108750501 podStartE2EDuration="2.108750501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:47.105675557 +0000 UTC m=+1147.173511125" watchObservedRunningTime="2026-10-30 19:49:47.108750501 +0000 UTC m=+1147.176586071" Oct 30 19:49:47.109401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.108881 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.109528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.109417 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmssh\" (UniqueName: \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.110569 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.110537 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.130194 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.130143 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podStartSLOduration=2.130128796 podStartE2EDuration="2.130128796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:47.129602969 +0000 UTC m=+1147.197438538" watchObservedRunningTime="2026-10-30 19:49:47.130128796 +0000 UTC m=+1147.197964366" Oct 30 19:49:47.131538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.131507 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmssh\" (UniqueName: \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\") pod \"payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.210027 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.209990 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:47.595656 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.595631 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 19:49:47.597618 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:47.597591 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88acb5d2_4a06_4b9f_a408_0137dad64956.slice/crio-498ac3bb8d0ab48ce716b6fccd470e0584900fc258af744d78629a1185e1f33a WatchSource:0}: Error finding container 498ac3bb8d0ab48ce716b6fccd470e0584900fc258af744d78629a1185e1f33a: Status 404 returned error can't find the container with id 498ac3bb8d0ab48ce716b6fccd470e0584900fc258af744d78629a1185e1f33a Oct 30 19:49:47.666445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.666420 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 19:49:47.669862 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.669839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.670544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.670530 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.682051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.682032 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-278ffa-dockercfg-cvh8s\"" Oct 30 19:49:47.690089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.686936 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-278ffa\"" Oct 30 19:49:47.690089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.689471 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 19:49:47.709228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.709204 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 19:49:47.714275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.714252 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.731991 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.731969 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 19:49:47.817911 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.817869 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.818069 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.817932 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.818069 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.818021 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkwsn\" (UniqueName: \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.818236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.818215 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v862h\" (UniqueName: \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.860224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.860144 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.919224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.919195 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v862h\" (UniqueName: \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.919365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.919270 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.919365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.919316 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.919365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.919348 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkwsn\" (UniqueName: \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.919983 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.919961 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.920131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.920114 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.936510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.936484 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkwsn\" (UniqueName: \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\") pod \"payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:47.938831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.938781 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v862h\" (UniqueName: \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\") pod \"payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:47.982943 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:47.982918 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:48.030103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.029266 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:48.064284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.062933 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:48.064284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.062978 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" event={"ID":"88acb5d2-4a06-4b9f-a408-0137dad64956","Type":"ContainerStarted","Data":"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b"} Oct 30 19:49:48.064284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.063004 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" event={"ID":"88acb5d2-4a06-4b9f-a408-0137dad64956","Type":"ContainerStarted","Data":"498ac3bb8d0ab48ce716b6fccd470e0584900fc258af744d78629a1185e1f33a"} Oct 30 19:49:48.117239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.117123 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podStartSLOduration=2.117107069 podStartE2EDuration="2.117107069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:48.1163986 +0000 UTC m=+1148.184234169" watchObservedRunningTime="2026-10-30 19:49:48.117107069 +0000 UTC m=+1148.184942636" Oct 30 19:49:48.170017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.169980 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.303528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.303502 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 19:49:48.304511 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:48.304485 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78000019_d14f_4c89_b1d4_06dfcbd4be4e.slice/crio-d105769dde36de69606017d8db29d8486dd51a3bb6750a667933f8249498da71 WatchSource:0}: Error finding container d105769dde36de69606017d8db29d8486dd51a3bb6750a667933f8249498da71: Status 404 returned error can't find the container with id d105769dde36de69606017d8db29d8486dd51a3bb6750a667933f8249498da71 Oct 30 19:49:48.399358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.399332 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 19:49:48.400997 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:48.400967 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bc38b8_b8b2_434e_819b_e7502f4d4441.slice/crio-e399d35c49300bfa2121e6eca4d5d408214c50def0bdbb492484788aff82ec37 WatchSource:0}: Error finding container e399d35c49300bfa2121e6eca4d5d408214c50def0bdbb492484788aff82ec37: Status 404 returned error can't find the container with id e399d35c49300bfa2121e6eca4d5d408214c50def0bdbb492484788aff82ec37 Oct 30 19:49:48.545736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.545706 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.813709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:48.813613 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.122076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.121522 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" event={"ID":"51bc38b8-b8b2-434e-819b-e7502f4d4441","Type":"ContainerStarted","Data":"140d65a139a52a8db1def49aa9fa17d8e3b0ae77326625834c9057e339e3174e"} Oct 30 19:49:49.122076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.121584 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" event={"ID":"51bc38b8-b8b2-434e-819b-e7502f4d4441","Type":"ContainerStarted","Data":"e399d35c49300bfa2121e6eca4d5d408214c50def0bdbb492484788aff82ec37"} Oct 30 19:49:49.122605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.122481 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:49:49.157096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.156157 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" event={"ID":"78000019-d14f-4c89-b1d4-06dfcbd4be4e","Type":"ContainerStarted","Data":"a1306238a7d548e81974eab2be415bc43ed2aa12c82ffbb815fde40b039e1ff6"} Oct 30 19:49:49.157096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.156312 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" event={"ID":"78000019-d14f-4c89-b1d4-06dfcbd4be4e","Type":"ContainerStarted","Data":"d105769dde36de69606017d8db29d8486dd51a3bb6750a667933f8249498da71"} Oct 30 19:49:49.157096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.156825 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:49:49.159624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.158686 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.208399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.208348 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podStartSLOduration=2.208333489 podStartE2EDuration="2.208333489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.206622801 +0000 UTC m=+1149.274458369" watchObservedRunningTime="2026-10-30 19:49:49.208333489 +0000 UTC m=+1149.276169056" Oct 30 19:49:49.209116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.209086 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podStartSLOduration=2.2090783099999998 podStartE2EDuration="2.20907831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:49.165976673 +0000 UTC m=+1149.233812242" watchObservedRunningTime="2026-10-30 19:49:49.20907831 +0000 UTC m=+1149.276913872" Oct 30 19:49:49.370963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.370927 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.911527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:49.911487 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.057166 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:50.057026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.252391 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:50.252288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.429007 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:50.428950 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.892789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:50.892754 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.038663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:51.038634 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.984802 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:51.984733 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.135749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.135716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.277485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.277396 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.909474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.909438 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 19:49:52.915644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.915620 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:52.932420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.932199 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-278ffa-serving-cert\"" Oct 30 19:49:52.932420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.932261 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 19:49:52.932420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.932356 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:49:52.932420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.932388 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\"" Oct 30 19:49:52.932713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.932440 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:49:52.938148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:52.934841 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 19:49:53.086119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.086047 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.086567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.086185 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tvdr\" (UniqueName: \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.086567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.086301 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.086567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.086337 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.110040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.110007 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.187398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.187310 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.187398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.187342 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.187629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.187451 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.187629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.187516 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tvdr\" (UniqueName: \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.188545 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.188517 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.190441 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.190412 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.190587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.190481 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.203802 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.203773 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tvdr\" (UniqueName: \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\") pod \"maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.238550 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.238510 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:53.336025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.335847 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.540036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:53.540002 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 19:49:53.540995 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:49:53.540967 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb50a54_7c85_49dc_abfd_d9234707efd1.slice/crio-7afda34ed5c50399ffff3001f21dabc7fc2132ae8818c00c4e6bf04f4e2f42a3 WatchSource:0}: Error finding container 7afda34ed5c50399ffff3001f21dabc7fc2132ae8818c00c4e6bf04f4e2f42a3: Status 404 returned error can't find the container with id 7afda34ed5c50399ffff3001f21dabc7fc2132ae8818c00c4e6bf04f4e2f42a3 Oct 30 19:49:54.190240 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:54.190195 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" event={"ID":"ddb50a54-7c85-49dc-abfd-d9234707efd1","Type":"ContainerStarted","Data":"7afda34ed5c50399ffff3001f21dabc7fc2132ae8818c00c4e6bf04f4e2f42a3"} Oct 30 19:49:54.483081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:54.482969 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:55.422614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:55.422494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.443106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:56.443010 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.624949 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:56.623838 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:57.055082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:57.048232 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 19:49:57.240133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:57.240096 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" event={"ID":"ddb50a54-7c85-49dc-abfd-d9234707efd1","Type":"ContainerStarted","Data":"3d859bfbec7b6698c0d1a91bc28867785e1cc0eb7b910ba21f19f31cad71d644"} Oct 30 19:49:57.248480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:57.241315 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:49:57.333106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:57.332992 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.061631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.061581 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.073825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.073798 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 19:49:58.074139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.074123 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 19:49:58.074303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.074291 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 19:49:58.074522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.074510 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 19:49:58.101782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.101735 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podStartSLOduration=3.240487322 podStartE2EDuration="6.101718594s" totalImagesPullingTime="2.861231272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:57.277021857 +0000 UTC m=+1157.344857425" watchObservedRunningTime="2026-10-30 19:49:58.101718594 +0000 UTC m=+1158.169554164" Oct 30 19:49:58.277297 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.277264 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.706278 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.706245 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.868510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:58.868478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.065398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.065204 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.182090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.181987 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 19:49:59.361159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.357666 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.437128 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:49:59.436247 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:49:59.678542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.678446 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.782987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.782957 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 19:49:59.788455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.788424 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.804436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.804395 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"simulator-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 19:49:59.804651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.804635 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"default-dockercfg-zwdzq\"" Oct 30 19:49:59.809900 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.809874 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"kube-root-ca.crt\"" Oct 30 19:49:59.810036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.809966 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"openshift-service-ca.crt\"" Oct 30 19:49:59.829721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.829682 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 19:49:59.886690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886542 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.886690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886666 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.886970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886739 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.886970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886779 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5z9b\" (UniqueName: \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.886970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886825 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.886970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.886857 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.939602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.939467 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.987768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.987729 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.987945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.987826 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.987945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.987893 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.987945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.987932 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5z9b\" (UniqueName: \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.988085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.988009 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.988085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.988077 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.988198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.988169 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.988322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.988299 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.988466 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.988445 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.990290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.990262 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:49:59.990775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:49:59.990753 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:00.004934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.004904 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5z9b\" (UniqueName: \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\") pod \"simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:00.112078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.110295 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:00.171319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.165213 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 19:50:00.187541 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.186947 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:00.541430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.530773 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 19:50:00.763603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:00.763558 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.005637 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.005591 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.170879 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.170845 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 19:50:01.252928 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.252087 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.360021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.359117 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" event={"ID":"5c087e21-e692-4eb2-8347-baa42ddc21d3","Type":"ContainerStarted","Data":"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9"} Oct 30 19:50:01.360021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.359169 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" event={"ID":"5c087e21-e692-4eb2-8347-baa42ddc21d3","Type":"ContainerStarted","Data":"8745f0fad1b36ab6a29d27ec423673a1476572015f49685fe839f2ecbd5cbc03"} Oct 30 19:50:01.437687 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:01.437322 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podStartSLOduration=2.437299841 podStartE2EDuration="2.437299841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:01.436319154 +0000 UTC m=+1161.504154725" watchObservedRunningTime="2026-10-30 19:50:01.437299841 +0000 UTC m=+1161.505135409" Oct 30 19:50:02.114231 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:02.114160 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:02.656813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:02.656774 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.539080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:03.538806 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.881846 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:03.881791 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.090496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:04.090459 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.296872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:04.296789 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 19:50:04.544933 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:04.544858 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.839753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:04.839663 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.060141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.060074 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.395128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.395028 5733 generic.go:356] "Generic (PLEG): container finished" podID="90effab3-a893-40cd-b620-673575f1d679" containerID="04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f" exitCode=0 Oct 30 19:50:05.395282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.395115 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerDied","Data":"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f"} Oct 30 19:50:05.395690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.395668 5733 scope.go:122] "RemoveContainer" containerID="04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f" Oct 30 19:50:05.397139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.397111 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:05.397230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.397175 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:05.397230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.397206 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:05.923609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:05.923547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:06.403520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.403489 5733 generic.go:356] "Generic (PLEG): container finished" podID="56d233c6-d1fc-4b59-8998-afe61144f065" containerID="30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd" exitCode=0 Oct 30 19:50:06.403963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.403571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerDied","Data":"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd"} Oct 30 19:50:06.404100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.404081 5733 scope.go:122] "RemoveContainer" containerID="30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd" Oct 30 19:50:06.405555 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.405529 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:06.405676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.405615 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:06.405676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.405667 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:06.406446 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.406422 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerStarted","Data":"107db65fd094c0b5f82e71abc806daeab1926735e830876b97614a7a65397855"} Oct 30 19:50:06.460881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.460846 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:50:06.461041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.461017 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:06.461127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.461107 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:06.895789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:06.895745 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.259145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:07.257870 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.441479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:07.440637 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerStarted","Data":"99161c544933d962fe7ffd9827eced0782c37807ef4eb3118699eb7dfb3af1a1"} Oct 30 19:50:07.469122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:07.468397 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:07.469122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:07.468470 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:08.092470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.092433 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:50:08.093807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.093469 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:50:08.093807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.093530 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 19:50:08.100882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.100770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:08.361355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.347463 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:08.467092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.464484 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:08.467092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:08.464537 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:09.041659 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.041619 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.090667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.090628 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:50:09.090971 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.090923 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 19:50:09.462890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.462445 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:09.462890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.462505 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:09.802027 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.801947 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.946686 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:09.946570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:10.091090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.091000 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:50:10.091310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.091288 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 19:50:10.125096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.119890 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:10.194406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.194367 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:10.203739 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.203705 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:10.466083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.460477 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:10.466083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.460540 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:10.499078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.499034 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:50:10.588749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.588699 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:10.882636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:10.882478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.087281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.087237 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.091290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.091263 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:50:11.091453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.091342 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 19:50:11.430672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.430595 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.472419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.471459 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 19:50:11.472419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.471522 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 19:50:11.982989 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.982952 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:50:11.983452 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:11.983039 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:50:12.092984 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.092945 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 19:50:12.093160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.093015 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 19:50:12.465092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.464921 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:50:12.505196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.505163 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 19:50:12.811147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.809879 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:12.995932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.995843 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:12.996479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:12.995935 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:13.375376 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.375343 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:50:13.377514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.377486 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 19:50:13.515514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.514618 5733 generic.go:356] "Generic (PLEG): container finished" podID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerID="7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1" exitCode=0 Oct 30 19:50:13.515514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.514692 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerDied","Data":"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1"} Oct 30 19:50:13.515514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.515340 5733 scope.go:122] "RemoveContainer" containerID="7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1" Oct 30 19:50:13.516950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.516916 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:13.517140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.517003 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:13.517140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.517071 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:50:13.821835 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:13.821736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.420777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:14.420649 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.540218 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:14.540185 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerStarted","Data":"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e"} Oct 30 19:50:14.836716 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:14.836615 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:50:14.836994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:14.836970 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:14.837173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:14.837149 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:15.049303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:15.049264 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.250552 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:15.250497 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.443296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:15.443260 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:15.836918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:15.836875 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:15.837135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:15.836953 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:16.202973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:16.202935 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.837548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:16.837509 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:16.838159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:16.838126 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:17.128547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:17.128511 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.840081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:17.839482 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:17.840081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:17.839549 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:18.146791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:18.146752 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:18.841620 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:18.841570 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 19:50:18.842224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:18.841626 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 19:50:19.175902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.175324 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:19.749877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.749843 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 19:50:19.754528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.754504 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.760435 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.760413 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"kube-root-ca.crt\"" Oct 30 19:50:19.764109 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.763656 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"openshift-service-ca.crt\"" Oct 30 19:50:19.766532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.766261 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"simulator-w6-5f933e-kserve-self-signed-certs\"" Oct 30 19:50:19.766532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.766326 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"default-dockercfg-dsn78\"" Oct 30 19:50:19.769829 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.769806 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 19:50:19.841695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.841664 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:50:19.842173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.842151 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:50:19.843146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.843127 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 19:50:19.852914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.852886 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:19.916083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.915756 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk5l\" (UniqueName: \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.916083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.915798 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.916354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.916009 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.916424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.916402 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.916481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.916440 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:19.916481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:19.916463 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018076 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018117 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018139 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018224 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk5l\" (UniqueName: \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018250 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018339 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.018750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.018727 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.019197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.019173 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.019469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.019449 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.021142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.021080 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.021769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.021746 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.051105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.051040 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk5l\" (UniqueName: \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\") pod \"simulator-w6-5f933e-kserve-86bf55fd5b-pncbq\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.071615 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.071577 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:20.115612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.115570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:20.403128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.403099 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 19:50:20.405284 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:50:20.405250 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c1b72f0_9a3d_41d2_863b_f8b2ed2b36c8.slice/crio-4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5 WatchSource:0}: Error finding container 4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5: Status 404 returned error can't find the container with id 4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5 Oct 30 19:50:20.609209 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.609167 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" event={"ID":"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8","Type":"ContainerStarted","Data":"88d28d0c138dc51caa10e6e00fb5090c2eccb2357b72774451571b2f3677e47c"} Oct 30 19:50:20.609922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.609896 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" event={"ID":"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8","Type":"ContainerStarted","Data":"4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5"} Oct 30 19:50:20.640395 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:20.640339 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podStartSLOduration=1.640325488 podStartE2EDuration="1.640325488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:20.638719329 +0000 UTC m=+1180.706554897" watchObservedRunningTime="2026-10-30 19:50:20.640325488 +0000 UTC m=+1180.708161056" Oct 30 19:50:21.072570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:21.072481 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:21.203594 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:21.203558 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.040233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:22.040201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.273441 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:22.273409 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.070552 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:23.070517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.841692 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:23.841652 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:24.883574 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:24.883467 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:25.889404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:25.889323 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.116812 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.116777 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.544502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.544410 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 19:50:27.550826 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.550797 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.558855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.558824 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 19:50:27.568477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.568454 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"premium-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 19:50:27.638308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.638271 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.708950 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.709029 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.709158 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.709255 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zsk\" (UniqueName: \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.709294 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.709915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.709322 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.778198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.778157 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.810634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810527 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.810791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810686 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zsk\" (UniqueName: \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.810791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810727 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.810791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810760 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.810964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810792 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.810964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810892 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.811101 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.810959 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.811101 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.811094 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.811299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.811281 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.813179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.813151 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.813452 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.813430 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.824213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.824187 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zsk\" (UniqueName: \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\") pod \"premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:27.868140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:27.868111 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:28.153685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.153653 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 19:50:28.155391 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:50:28.155362 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30e3bfe8_a903_4119_b1a6_46417f98910f.slice/crio-1c75248c4e6cc901caa967f5905afc2bd1066a44e21c9c4a94c8377877ddcd3a WatchSource:0}: Error finding container 1c75248c4e6cc901caa967f5905afc2bd1066a44e21c9c4a94c8377877ddcd3a: Status 404 returned error can't find the container with id 1c75248c4e6cc901caa967f5905afc2bd1066a44e21c9c4a94c8377877ddcd3a Oct 30 19:50:28.506694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.506604 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:28.666434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.666389 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" event={"ID":"30e3bfe8-a903-4119-b1a6-46417f98910f","Type":"ContainerStarted","Data":"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0"} Oct 30 19:50:28.666596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.666445 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" event={"ID":"30e3bfe8-a903-4119-b1a6-46417f98910f","Type":"ContainerStarted","Data":"1c75248c4e6cc901caa967f5905afc2bd1066a44e21c9c4a94c8377877ddcd3a"} Oct 30 19:50:28.687753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.687703 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podStartSLOduration=1.6876904050000001 podStartE2EDuration="1.687690405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:28.686913295 +0000 UTC m=+1188.754748865" watchObservedRunningTime="2026-10-30 19:50:28.687690405 +0000 UTC m=+1188.755525974" Oct 30 19:50:28.868593 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:28.868562 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:29.377767 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:50:29.377733 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:50:29.407429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:29.407388 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:29.859012 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:29.858970 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.071731 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:30.071696 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:30.084556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:30.084527 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:30.190433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:30.190347 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.688619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:30.688591 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:50:31.022306 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:31.022208 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.878808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:31.878770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.780489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:32.780453 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.358183 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:33.358142 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.710620 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:33.710535 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.592362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:34.592321 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:35.387302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.386471 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:35.463942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.463910 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 19:50:35.468393 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.468366 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.478820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.478796 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"kube-root-ca.crt\"" Oct 30 19:50:35.478820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.478796 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"openshift-service-ca.crt\"" Oct 30 19:50:35.479030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.478796 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"premium-w4-3f57c5-kserve-self-signed-certs\"" Oct 30 19:50:35.479030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.478796 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"default-dockercfg-6842t\"" Oct 30 19:50:35.481899 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.481871 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 19:50:35.574232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.574202 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:35.609580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.609546 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.610190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.610169 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.610575 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.610254 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.610575 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.610407 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.610575 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.610488 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.610575 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.610560 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6lh\" (UniqueName: \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.711508 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711426 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.711508 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711482 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711545 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711615 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6lh\" (UniqueName: \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711716 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711745 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711929 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.711985 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.712401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.712128 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.714336 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.714309 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.714471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.714447 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.728699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.728668 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6lh\" (UniqueName: \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\") pod \"premium-w4-3f57c5-kserve-c49958c8f-85rhc\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:35.783610 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:35.783571 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:36.067451 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.067420 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 19:50:36.069126 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:50:36.069095 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e76d74_7775_4b7b_9120_885c993ad626.slice/crio-c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2 WatchSource:0}: Error finding container c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2: Status 404 returned error can't find the container with id c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2 Oct 30 19:50:36.185526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.185494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:36.716355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.716323 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" event={"ID":"86e76d74-7775-4b7b-9120-885c993ad626","Type":"ContainerStarted","Data":"bdfa3d6a341557552c83656e814a421ff9e5c0179cc42cc26ac9de55cc54af46"} Oct 30 19:50:36.716748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.716362 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" event={"ID":"86e76d74-7775-4b7b-9120-885c993ad626","Type":"ContainerStarted","Data":"c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2"} Oct 30 19:50:36.748989 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.748936 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podStartSLOduration=1.748920676 podStartE2EDuration="1.748920676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:36.738321943 +0000 UTC m=+1196.806157512" watchObservedRunningTime="2026-10-30 19:50:36.748920676 +0000 UTC m=+1196.816756290" Oct 30 19:50:36.784216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:36.784183 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:37.158716 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:37.158683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:37.868750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:37.868713 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:37.881596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:37.881571 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:38.108207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:38.108175 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.391080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:38.391021 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.749173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:38.749078 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:50:38.882176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:38.882143 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.145893 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:39.145836 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.787194 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:39.787158 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.771649 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:40.771599 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:41.580719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:41.580675 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.145744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:42.145709 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.499131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:42.499037 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:43.244009 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:43.243979 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:44.014250 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:44.014211 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:44.787270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:44.787234 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.669651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:45.669612 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.784134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:45.784103 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:45.797986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:45.797954 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:46.431379 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:46.431341 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:46.795849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:46.795768 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:50:47.220349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:47.220257 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.167554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:48.167525 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.483622 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:48.483517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:49.003621 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:49.003570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.162078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:50.161420 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.364086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:50.363153 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:51.091217 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:51.090544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.069081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:52.068239 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.891277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:52.891246 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:53.737001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:53.736313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:54.695329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:54.695290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.488490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:55.488437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.821287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:55.821206 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:56.518641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:56.518606 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:57.473774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:57.473738 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:58.766295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:58.766257 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:59.379844 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:50:59.379804 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:50:59.703434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:50:59.703303 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:00.542990 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:00.542950 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:01.321081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:01.321013 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:02.275929 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:02.275885 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:03.106389 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:03.106346 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:03.900540 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:03.900494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:04.962900 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:04.962833 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.209801 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:05.209749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.422038 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:05.421995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.905166 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:05.905128 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.816934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:06.816897 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.821674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:07.821635 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:09.053554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:09.053522 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.120534 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.120500 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 19:51:10.125228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.125205 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.141893 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.141871 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ced832-openshift-default-dockercfg-4thzg\"" Oct 30 19:51:10.147502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147474 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147535 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147674 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147782 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147854 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147923 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147896 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x98l\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.147982 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.147966 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.148023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.148011 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.148086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.148074 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.162853 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.162826 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 19:51:10.249520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249480 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249546 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249583 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249607 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249638 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x98l\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249685 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249717 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.249780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249751 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.250217 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.249853 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.250473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.250443 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.250683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.250660 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.250858 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.250834 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.251181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.251159 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.251394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.251365 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.252350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.252326 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.253409 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.253386 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.268401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.268371 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x98l\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.280286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.280263 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.449808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.446565 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.449808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.449270 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:10.874032 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.873995 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 19:51:10.881555 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:10.881519 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8878558a_bcd4_478e_a62e_3edcd26ab88f.slice/crio-1e080b077e9ecdce444c65dc904be0de8674e13456c178c28f3692489e5de98d WatchSource:0}: Error finding container 1e080b077e9ecdce444c65dc904be0de8674e13456c178c28f3692489e5de98d: Status 404 returned error can't find the container with id 1e080b077e9ecdce444c65dc904be0de8674e13456c178c28f3692489e5de98d Oct 30 19:51:10.883974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.883937 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:10.884139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.884020 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:10.884139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:10.884114 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:11.009904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.009695 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" event={"ID":"8878558a-bcd4-478e-a62e-3edcd26ab88f","Type":"ContainerStarted","Data":"583bed1fc60a3b586e96e7a3303fa1efdfa4cdab0937eca1f6786528ecfcce52"} Oct 30 19:51:11.009904 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.009746 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" event={"ID":"8878558a-bcd4-478e-a62e-3edcd26ab88f","Type":"ContainerStarted","Data":"1e080b077e9ecdce444c65dc904be0de8674e13456c178c28f3692489e5de98d"} Oct 30 19:51:11.043820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.043664 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podStartSLOduration=1.043644998 podStartE2EDuration="1.043644998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:11.041858924 +0000 UTC m=+1231.109694494" watchObservedRunningTime="2026-10-30 19:51:11.043644998 +0000 UTC m=+1231.111480567" Oct 30 19:51:11.452248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.452216 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:11.453813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.453782 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:11.453968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.453841 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:11.504603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:11.504566 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.452453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:12.452413 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:12.453125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:12.453086 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:12.519976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:12.519941 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.289527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:13.289492 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.452343 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:13.452304 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:13.452519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:13.452384 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:14.088045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:14.087953 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:14.451442 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:14.451403 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:14.451627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:14.451473 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:14.922082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:14.920582 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.452100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:15.452038 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.452272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:15.452129 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:15.736896 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:15.736755 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:16.451559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:16.451517 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.451939 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:16.451585 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:16.872916 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:16.872885 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:17.452438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:17.452399 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.452831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:17.452475 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 30 19:51:17.862126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:17.862037 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:18.456694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:18.456661 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:18.457190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:18.457086 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:18.457688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:18.457667 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 19:51:18.887541 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:18.887505 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:19.893950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:19.893900 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.810532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:20.810445 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:21.691611 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:21.691570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:22.490134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:22.490101 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:23.541076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:23.541025 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:24.451617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.451582 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:24.782897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.782806 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 19:51:24.787333 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.787308 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.801199 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.801174 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9\"" Oct 30 19:51:24.813855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.813821 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 19:51:24.964037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.963999 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq665\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964037 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964051 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964147 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964193 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964227 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964250 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964279 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964306 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:24.964512 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:24.964435 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.066249 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.066157 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.066932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.066905 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.067570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.067547 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq665\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.067742 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.067726 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.067865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.067852 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.067991 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068135 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068265 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068251 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068383 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068370 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068601 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068785 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.068881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.068845 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.069756 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.069733 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.070198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.070177 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.074072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.072331 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.074072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.072781 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.088312 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.088285 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq665\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.091338 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.091315 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.103337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.103299 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:25.600128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.597144 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 19:51:25.606095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.605125 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:25.610034 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.609987 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:25.610195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:25.610142 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:26.070302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.063827 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 19:51:26.070984 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.070933 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.078724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.078693 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6\"" Oct 30 19:51:26.098671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.098633 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 19:51:26.169077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.168626 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:26.169077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.168819 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" event={"ID":"a74bb1f9-6fa8-4a48-92ef-62b50d195958","Type":"ContainerStarted","Data":"9d764348a646ef3fb5c03204bcb5098f3c795e6202ed2f6b595e236738edb827"} Oct 30 19:51:26.169077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.168868 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" event={"ID":"a74bb1f9-6fa8-4a48-92ef-62b50d195958","Type":"ContainerStarted","Data":"1e65fb432832c403d4164a966ba85ba67b78264adc608aec31202897216b1a22"} Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.229959 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230078 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kgl\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230146 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230174 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230297 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230329 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230434 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230461 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.232082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.230522 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.331706 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.331772 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.331893 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.331920 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.331975 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.332020 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.332127 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kgl\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.332173 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.332202 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.333219 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.333944 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.335010 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335850 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.335302 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.335850 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.335534 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.342089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.340755 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.344454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.344427 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.357624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.357594 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kgl\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.358010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.357982 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.408459 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.408133 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:26.805804 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.805747 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podStartSLOduration=2.805728745 podStartE2EDuration="2.805728745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:26.223092332 +0000 UTC m=+1246.290927902" watchObservedRunningTime="2026-10-30 19:51:26.805728745 +0000 UTC m=+1246.873564312" Oct 30 19:51:26.805997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.805977 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 19:51:26.806592 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:26.806564 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51892042_d2ae_4a69_9cc4_e347399249ed.slice/crio-a57538f1222e3b32814363675a0747dc15b655e4234743478f7dab48ce935eab WatchSource:0}: Error finding container a57538f1222e3b32814363675a0747dc15b655e4234743478f7dab48ce935eab: Status 404 returned error can't find the container with id a57538f1222e3b32814363675a0747dc15b655e4234743478f7dab48ce935eab Oct 30 19:51:26.814076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.810671 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:26.814076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.810760 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:26.814076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:26.810809 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:27.104467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.104436 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:27.105834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.105586 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:27.105834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.105628 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:27.177009 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.176971 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" event={"ID":"51892042-d2ae-4a69-9cc4-e347399249ed","Type":"ContainerStarted","Data":"971c24d0caa4087ea0b18f3e1a5fed49f146c79ec5060061df0705ba51390b26"} Oct 30 19:51:27.177210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.177037 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" event={"ID":"51892042-d2ae-4a69-9cc4-e347399249ed","Type":"ContainerStarted","Data":"a57538f1222e3b32814363675a0747dc15b655e4234743478f7dab48ce935eab"} Oct 30 19:51:27.224081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.217935 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podStartSLOduration=1.217913862 podStartE2EDuration="1.217913862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:27.209680953 +0000 UTC m=+1247.277516546" watchObservedRunningTime="2026-10-30 19:51:27.217913862 +0000 UTC m=+1247.285749431" Oct 30 19:51:27.409535 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.409457 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:27.420082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.412703 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:27.420082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.412758 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:27.534101 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:27.532048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.104047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:28.104019 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:28.104301 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:28.104277 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:28.409476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:28.409381 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:28.409476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:28.409442 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:28.488993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:28.488759 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:29.087378 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.087341 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c"] Oct 30 19:51:29.091245 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.091222 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.098119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.098087 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796tf\" (UniqueName: \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.098368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.098348 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.098457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.098387 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.100306 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.100285 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:51:29.100437 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.100414 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:29.104191 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.104168 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:29.104347 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.104324 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:29.107088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.106233 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c"] Oct 30 19:51:29.204261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.199214 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.204261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.199245 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.204261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.199278 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796tf\" (UniqueName: \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.204261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.200552 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.204261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.201892 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.253080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.250807 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796tf\" (UniqueName: \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\") pod \"maas-api-revoke-sub-6bc9b3d5-gq84c\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.392091 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:51:29.390675 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:51:29.411525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.411247 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:29.411525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.411466 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:29.411525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.411517 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:29.419085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.416422 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:29.882089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.881473 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c"] Oct 30 19:51:29.882661 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:29.882629 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode985d03f_0643_472c_ab7f_d0cb147a2636.slice/crio-712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17 WatchSource:0}: Error finding container 712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17: Status 404 returned error can't find the container with id 712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17 Oct 30 19:51:29.884653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:29.884636 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:51:30.104510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.104475 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:30.104673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.104530 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:30.197096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.197004 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" event={"ID":"e985d03f-0643-472c-ab7f-d0cb147a2636","Type":"ContainerStarted","Data":"712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17"} Oct 30 19:51:30.408004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.407842 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:30.414118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.410213 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:30.414118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:30.410275 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:31.104570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:31.104530 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:31.104738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:31.104592 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:31.325092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:31.324996 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.409707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:31.409622 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:31.409707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:31.409688 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:32.111083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:32.104795 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 19:51:32.111083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:32.104854 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 19:51:32.409227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:32.409138 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:32.409227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:32.409203 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:32.454082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:32.453823 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:33.110355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.110315 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:33.111016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.110992 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:33.111905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.111879 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 19:51:33.366514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.366433 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:33.408963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.408927 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 19:51:33.409179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:33.408988 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 19:51:34.230298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:34.230252 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.414429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:34.414392 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:34.414776 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:34.414749 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:34.415549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:34.415525 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 19:51:35.249331 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:35.248078 5733 generic.go:356] "Generic (PLEG): container finished" podID="e985d03f-0643-472c-ab7f-d0cb147a2636" containerID="d4fedf545716c71fff73f7d73287b0eeb7542bdff217625fe1c525aff04f52de" exitCode=0 Oct 30 19:51:35.249331 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:35.248159 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" event={"ID":"e985d03f-0643-472c-ab7f-d0cb147a2636","Type":"ContainerDied","Data":"d4fedf545716c71fff73f7d73287b0eeb7542bdff217625fe1c525aff04f52de"} Oct 30 19:51:35.563662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:35.563570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:36.439016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.438981 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf"] Oct 30 19:51:36.444076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.444037 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.462831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.462805 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf"] Oct 30 19:51:36.463987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.463965 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:36.563932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.563837 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\") pod \"e985d03f-0643-472c-ab7f-d0cb147a2636\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " Oct 30 19:51:36.563932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.563909 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\" (UniqueName: \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\") pod \"e985d03f-0643-472c-ab7f-d0cb147a2636\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " Oct 30 19:51:36.564185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564013 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\") pod \"e985d03f-0643-472c-ab7f-d0cb147a2636\" (UID: \"e985d03f-0643-472c-ab7f-d0cb147a2636\") " Oct 30 19:51:36.564310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564282 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca" pod "e985d03f-0643-472c-ab7f-d0cb147a2636" (UID: "e985d03f-0643-472c-ab7f-d0cb147a2636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:36.564392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564369 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.564476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564454 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.564597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564574 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxpp6\" (UniqueName: \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.564768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.564741 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:36.566911 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.566885 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca" pod "e985d03f-0643-472c-ab7f-d0cb147a2636" (UID: "e985d03f-0643-472c-ab7f-d0cb147a2636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:36.573284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.573254 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf" pod "e985d03f-0643-472c-ab7f-d0cb147a2636" (UID: "e985d03f-0643-472c-ab7f-d0cb147a2636"). InnerVolumeSpecName "kube-api-access-796tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:36.666114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666049 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxpp6\" (UniqueName: \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.666286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666223 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.666286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666244 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.666405 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666293 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796tf\" (UniqueName: \"kubernetes.io/projected/e985d03f-0643-472c-ab7f-d0cb147a2636-kube-api-access-796tf\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:36.666405 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666308 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e985d03f-0643-472c-ab7f-d0cb147a2636-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:36.666961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.666935 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.668560 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.668540 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.686859 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.686833 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxpp6\" (UniqueName: \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\") pod \"maas-api-revoke-sub-6e2166da-q5vrf\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:36.765793 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:36.765520 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:37.078837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.078757 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.091470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.091432 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf"] Oct 30 19:51:37.095421 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:37.095394 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcccf22c_5864_408d_b806_0dd13fba3cff.slice/crio-fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf WatchSource:0}: Error finding container fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf: Status 404 returned error can't find the container with id fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf Oct 30 19:51:37.265618 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.265589 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" Oct 30 19:51:37.265765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.265631 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c" event={"ID":"e985d03f-0643-472c-ab7f-d0cb147a2636","Type":"ContainerDied","Data":"712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17"} Oct 30 19:51:37.265765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.265691 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="712556168d4034d8d51db753fb5cca6d47f100c5d6d95bf94c92153bece27d17" Oct 30 19:51:37.267205 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:37.267180 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" event={"ID":"fcccf22c-5864-408d-b806-0dd13fba3cff","Type":"ContainerStarted","Data":"fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf"} Oct 30 19:51:38.102796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:38.102762 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:38.274447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:38.274366 5733 generic.go:356] "Generic (PLEG): container finished" podID="fcccf22c-5864-408d-b806-0dd13fba3cff" containerID="46731d708d094bbb7759291f1b60400e0aee10244f9ed7f374797964c608ae40" exitCode=0 Oct 30 19:51:38.274590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:38.274447 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" event={"ID":"fcccf22c-5864-408d-b806-0dd13fba3cff","Type":"ContainerDied","Data":"46731d708d094bbb7759291f1b60400e0aee10244f9ed7f374797964c608ae40"} Oct 30 19:51:38.920081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:38.917968 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:39.450224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.450200 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:39.610111 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.610045 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\") pod \"fcccf22c-5864-408d-b806-0dd13fba3cff\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " Oct 30 19:51:39.610272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.610168 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\") pod \"fcccf22c-5864-408d-b806-0dd13fba3cff\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " Oct 30 19:51:39.610272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.610212 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\" (UniqueName: \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\") pod \"fcccf22c-5864-408d-b806-0dd13fba3cff\" (UID: \"fcccf22c-5864-408d-b806-0dd13fba3cff\") " Oct 30 19:51:39.610572 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.610543 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca" pod "fcccf22c-5864-408d-b806-0dd13fba3cff" (UID: "fcccf22c-5864-408d-b806-0dd13fba3cff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:39.610807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.610787 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:39.612145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.612113 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca" pod "fcccf22c-5864-408d-b806-0dd13fba3cff" (UID: "fcccf22c-5864-408d-b806-0dd13fba3cff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:39.612384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.612359 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6" pod "fcccf22c-5864-408d-b806-0dd13fba3cff" (UID: "fcccf22c-5864-408d-b806-0dd13fba3cff"). InnerVolumeSpecName "kube-api-access-dxpp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:39.711890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.711859 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcccf22c-5864-408d-b806-0dd13fba3cff-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:39.711890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.711888 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxpp6\" (UniqueName: \"kubernetes.io/projected/fcccf22c-5864-408d-b806-0dd13fba3cff-kube-api-access-dxpp6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:39.738250 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:39.738166 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.290545 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.290516 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" Oct 30 19:51:40.290545 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.290538 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf" event={"ID":"fcccf22c-5864-408d-b806-0dd13fba3cff","Type":"ContainerDied","Data":"fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf"} Oct 30 19:51:40.290766 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.290579 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc83d2a9e39d778cfb8eaf291de4ecf0645c1f9dd63085abe496092956b34fdf" Oct 30 19:51:40.604237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604202 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 19:51:40.604719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604706 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e985d03f-0643-472c-ab7f-d0cb147a2636" containerName="revoke-keys" Oct 30 19:51:40.604778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604720 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e985d03f-0643-472c-ab7f-d0cb147a2636" containerName="revoke-keys" Oct 30 19:51:40.604778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604752 5733 cpu_manager.go:414] "containerMap: removing container" podUID="fcccf22c-5864-408d-b806-0dd13fba3cff" containerName="revoke-keys" Oct 30 19:51:40.604778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604760 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcccf22c-5864-408d-b806-0dd13fba3cff" containerName="revoke-keys" Oct 30 19:51:40.604882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604872 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e985d03f-0643-472c-ab7f-d0cb147a2636" containerName="revoke-keys" Oct 30 19:51:40.604917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.604887 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcccf22c-5864-408d-b806-0dd13fba3cff" containerName="revoke-keys" Oct 30 19:51:40.608431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.608413 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.627458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.627425 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-278ffa-openshift-default-dockercfg-9bwm7\"" Oct 30 19:51:40.645461 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.645418 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 19:51:40.726467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726424 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hk2v\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726479 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726599 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726645 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726728 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726752 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726774 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.726917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.726907 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.727024 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.727000 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828456 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828522 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hk2v\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828545 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828603 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828642 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828698 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828721 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828750 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.828908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.828821 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.829131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.829087 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.829269 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.829247 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.829409 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.829382 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.829532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.829413 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.829532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.829479 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.830926 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.830906 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.831290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.831273 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.862688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.862626 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hk2v\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.863846 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.863825 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:40.897877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.897845 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.922230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:40.922207 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:41.220622 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.220593 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 19:51:41.222265 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:41.222236 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf117faf1_f824_4851_9ce6_e85ebd8fa690.slice/crio-462d8d62658f7c098d58af4be491731d9d362d1a80ed7baaf41c3431c0fa7f2b WatchSource:0}: Error finding container 462d8d62658f7c098d58af4be491731d9d362d1a80ed7baaf41c3431c0fa7f2b: Status 404 returned error can't find the container with id 462d8d62658f7c098d58af4be491731d9d362d1a80ed7baaf41c3431c0fa7f2b Oct 30 19:51:41.225340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.225301 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:41.225450 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.225382 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:41.225450 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.225437 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:51:41.304018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.298877 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" event={"ID":"f117faf1-f824-4851-9ce6-e85ebd8fa690","Type":"ContainerStarted","Data":"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de"} Oct 30 19:51:41.304018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.298950 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" event={"ID":"f117faf1-f824-4851-9ce6-e85ebd8fa690","Type":"ContainerStarted","Data":"462d8d62658f7c098d58af4be491731d9d362d1a80ed7baaf41c3431c0fa7f2b"} Oct 30 19:51:41.339468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.339399 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podStartSLOduration=1.339377113 podStartE2EDuration="1.339377113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:41.334736723 +0000 UTC m=+1261.402572295" watchObservedRunningTime="2026-10-30 19:51:41.339377113 +0000 UTC m=+1261.407212681" Oct 30 19:51:41.829102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.827232 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:41.923207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:41.923165 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:42.822566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:42.822526 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.933201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:42.933162 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:51:42.934166 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:42.934132 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:51:43.926502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:43.926466 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:51:43.926750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:43.926723 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:51:44.001161 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:44.001134 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.927079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:44.926441 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:51:44.927079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:44.926520 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:51:45.071778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.071740 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:45.147226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.147194 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv"] Oct 30 19:51:45.152624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.152585 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv"] Oct 30 19:51:45.152791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.152699 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.162422 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.162338 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:45.164948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.164550 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:51:45.197078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.196986 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.197230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.197146 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.197301 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.197248 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vljcl\" (UniqueName: \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.298400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.298367 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.298606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.298412 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.298606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.298494 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vljcl\" (UniqueName: \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.299491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.299463 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.301486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.301407 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.330083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.325026 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vljcl\" (UniqueName: \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\") pod \"maas-api-revoke-sub-e56dbeb5-k69vv\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.470507 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.470421 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:45.803742 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.803716 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv"] Oct 30 19:51:45.936117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.929158 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:51:45.936117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:45.929217 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:51:46.036443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.036407 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:46.360672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.360626 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" event={"ID":"f3e86dcc-ee0a-49c8-85f3-4c984fd07004","Type":"ContainerStarted","Data":"5aa2458f9fc1c8b41ce638f6ec4488278e0f3c3811a8db1a25a6e8f3f03dd9a4"} Oct 30 19:51:46.361202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.360687 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" event={"ID":"f3e86dcc-ee0a-49c8-85f3-4c984fd07004","Type":"ContainerStarted","Data":"0b1dbd6eabcc549209a744389ea0a60f0c82946d9a9c663baa0bddef30aa26a9"} Oct 30 19:51:46.392808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.392754 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" podStartSLOduration=1.392739974 podStartE2EDuration="1.392739974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:46.389081451 +0000 UTC m=+1266.456917013" watchObservedRunningTime="2026-10-30 19:51:46.392739974 +0000 UTC m=+1266.460575542" Oct 30 19:51:46.936576 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.929130 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 30 19:51:46.936576 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:46.929187 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 30 19:51:47.009586 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:47.009545 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:47.372789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:47.372742 5733 generic.go:356] "Generic (PLEG): container finished" podID="f3e86dcc-ee0a-49c8-85f3-4c984fd07004" containerID="5aa2458f9fc1c8b41ce638f6ec4488278e0f3c3811a8db1a25a6e8f3f03dd9a4" exitCode=0 Oct 30 19:51:47.373330 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:47.372947 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" event={"ID":"f3e86dcc-ee0a-49c8-85f3-4c984fd07004","Type":"ContainerDied","Data":"5aa2458f9fc1c8b41ce638f6ec4488278e0f3c3811a8db1a25a6e8f3f03dd9a4"} Oct 30 19:51:47.836667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:47.836552 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.140513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.140472 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:48.142585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.142521 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:48.151613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.151581 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 19:51:48.653471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.653441 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:48.702974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.702880 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.706570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.706542 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\") pod \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " Oct 30 19:51:48.706733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.706591 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\" (UniqueName: \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\") pod \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " Oct 30 19:51:48.706733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.706624 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\") pod \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\" (UID: \"f3e86dcc-ee0a-49c8-85f3-4c984fd07004\") " Oct 30 19:51:48.707464 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.707429 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca" pod "f3e86dcc-ee0a-49c8-85f3-4c984fd07004" (UID: "f3e86dcc-ee0a-49c8-85f3-4c984fd07004"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:48.708792 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.708761 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca" pod "f3e86dcc-ee0a-49c8-85f3-4c984fd07004" (UID: "f3e86dcc-ee0a-49c8-85f3-4c984fd07004"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:48.712510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.712480 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl" pod "f3e86dcc-ee0a-49c8-85f3-4c984fd07004" (UID: "f3e86dcc-ee0a-49c8-85f3-4c984fd07004"). InnerVolumeSpecName "kube-api-access-vljcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:48.808539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.808507 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.808539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.808535 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vljcl\" (UniqueName: \"kubernetes.io/projected/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-kube-api-access-vljcl\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.808539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:48.808548 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e86dcc-ee0a-49c8-85f3-4c984fd07004-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:49.417113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:49.415575 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" Oct 30 19:51:49.417113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:49.415609 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv" event={"ID":"f3e86dcc-ee0a-49c8-85f3-4c984fd07004","Type":"ContainerDied","Data":"0b1dbd6eabcc549209a744389ea0a60f0c82946d9a9c663baa0bddef30aa26a9"} Oct 30 19:51:49.417113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:49.415657 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1dbd6eabcc549209a744389ea0a60f0c82946d9a9c663baa0bddef30aa26a9" Oct 30 19:51:49.827019 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:49.826922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:50.632536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:50.632500 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.244084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:51.237493 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:52.306070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:52.306018 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:53.130961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.130923 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm"] Oct 30 19:51:53.131497 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.131482 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f3e86dcc-ee0a-49c8-85f3-4c984fd07004" containerName="revoke-keys" Oct 30 19:51:53.131560 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.131498 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e86dcc-ee0a-49c8-85f3-4c984fd07004" containerName="revoke-keys" Oct 30 19:51:53.131597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.131592 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e86dcc-ee0a-49c8-85f3-4c984fd07004" containerName="revoke-keys" Oct 30 19:51:53.135329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.135313 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.142133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.142108 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:53.142308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.142157 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:51:53.145735 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.145468 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm"] Oct 30 19:51:53.196745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.196711 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.196897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.196867 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwz5\" (UniqueName: \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.196940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.196908 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.219669 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.219634 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:53.298188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.298152 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwz5\" (UniqueName: \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.298354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.298197 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.298354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.298282 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.298827 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.298806 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.300527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.300505 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.312164 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.312136 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwz5\" (UniqueName: \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\") pod \"maas-api-revoke-sub-28dac7f8-88vmm\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.462751 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.462659 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:53.756609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:53.756576 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm"] Oct 30 19:51:53.757438 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:51:53.757408 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341bb8d5_89c5_44b2_9a6b_0e97287134f7.slice/crio-ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc WatchSource:0}: Error finding container ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc: Status 404 returned error can't find the container with id ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc Oct 30 19:51:54.090866 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:54.090826 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:54.490374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:54.490342 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" event={"ID":"341bb8d5-89c5-44b2-9a6b-0e97287134f7","Type":"ContainerStarted","Data":"53b9ec62a39c0b9fc9e3eadb360f823dc42cf1958a6f2e21624b87bc50e8f1d8"} Oct 30 19:51:54.490869 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:54.490397 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" event={"ID":"341bb8d5-89c5-44b2-9a6b-0e97287134f7","Type":"ContainerStarted","Data":"ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc"} Oct 30 19:51:54.514930 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:54.514873 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" podStartSLOduration=1.514858338 podStartE2EDuration="1.514858338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:54.514050525 +0000 UTC m=+1274.581886094" watchObservedRunningTime="2026-10-30 19:51:54.514858338 +0000 UTC m=+1274.582693906" Oct 30 19:51:54.913574 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:54.913535 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:55.498257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:55.498220 5733 generic.go:356] "Generic (PLEG): container finished" podID="341bb8d5-89c5-44b2-9a6b-0e97287134f7" containerID="53b9ec62a39c0b9fc9e3eadb360f823dc42cf1958a6f2e21624b87bc50e8f1d8" exitCode=0 Oct 30 19:51:55.498688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:55.498316 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" event={"ID":"341bb8d5-89c5-44b2-9a6b-0e97287134f7","Type":"ContainerDied","Data":"53b9ec62a39c0b9fc9e3eadb360f823dc42cf1958a6f2e21624b87bc50e8f1d8"} Oct 30 19:51:55.926109 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:55.926048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:56.464326 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.464290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:56.712834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.712807 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:56.862624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.862589 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\" (UniqueName: \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\") pod \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " Oct 30 19:51:56.862825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.862744 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\") pod \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " Oct 30 19:51:56.862898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.862825 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\") pod \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\" (UID: \"341bb8d5-89c5-44b2-9a6b-0e97287134f7\") " Oct 30 19:51:56.863272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.863236 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca" pod "341bb8d5-89c5-44b2-9a6b-0e97287134f7" (UID: "341bb8d5-89c5-44b2-9a6b-0e97287134f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:56.863413 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.863344 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.864905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.864877 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5" pod "341bb8d5-89c5-44b2-9a6b-0e97287134f7" (UID: "341bb8d5-89c5-44b2-9a6b-0e97287134f7"). InnerVolumeSpecName "kube-api-access-dxwz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:56.864905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.864900 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca" pod "341bb8d5-89c5-44b2-9a6b-0e97287134f7" (UID: "341bb8d5-89c5-44b2-9a6b-0e97287134f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:56.964784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.964742 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/341bb8d5-89c5-44b2-9a6b-0e97287134f7-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:56.964784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:56.964777 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwz5\" (UniqueName: \"kubernetes.io/projected/341bb8d5-89c5-44b2-9a6b-0e97287134f7-kube-api-access-dxwz5\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:57.088595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:57.088557 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:57.519249 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:57.519214 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" Oct 30 19:51:57.519430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:57.519214 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm" event={"ID":"341bb8d5-89c5-44b2-9a6b-0e97287134f7","Type":"ContainerDied","Data":"ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc"} Oct 30 19:51:57.519430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:57.519338 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab41ff87e1ff599bdd5f3ab56773c13008ddeee4ac7e8b03444ec46e073d5dfc" Oct 30 19:51:58.109951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:58.109914 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:59.377501 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:51:59.377465 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:51:59.394453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:51:59.394422 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:00.392203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:00.392165 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.441541 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:01.441510 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.396355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:02.396302 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.802080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:03.801966 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:04.976459 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:04.976426 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:06.442473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.442432 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:06.972941 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.972907 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l"] Oct 30 19:52:06.975123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.975093 5733 cpu_manager.go:414] "containerMap: removing container" podUID="341bb8d5-89c5-44b2-9a6b-0e97287134f7" containerName="revoke-keys" Oct 30 19:52:06.975383 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.975366 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="341bb8d5-89c5-44b2-9a6b-0e97287134f7" containerName="revoke-keys" Oct 30 19:52:06.975616 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.975601 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="341bb8d5-89c5-44b2-9a6b-0e97287134f7" containerName="revoke-keys" Oct 30 19:52:06.984701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.983541 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l"] Oct 30 19:52:06.984701 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.983634 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:06.999241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:06.999220 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:07.000168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.000133 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:52:07.176239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.176202 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cctq\" (UniqueName: \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.176433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.176264 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.176433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.176421 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.277471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.277381 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cctq\" (UniqueName: \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.277471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.277450 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.277656 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.277522 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.278175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.278149 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.279706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.279686 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.291384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.291363 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cctq\" (UniqueName: \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\") pod \"maas-api-revoke-sub-0d0e6762-68p2l\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.301211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.301189 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:07.559559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.559472 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:07.611385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.611357 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l"] Oct 30 19:52:07.612574 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:07.612541 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d9f3fd_024d_4e24_97fa_cea7523aea17.slice/crio-a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f WatchSource:0}: Error finding container a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f: Status 404 returned error can't find the container with id a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f Oct 30 19:52:07.650533 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:07.650503 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" event={"ID":"e0d9f3fd-024d-4e24-97fa-cea7523aea17","Type":"ContainerStarted","Data":"a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f"} Oct 30 19:52:08.659003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:08.658965 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" event={"ID":"e0d9f3fd-024d-4e24-97fa-cea7523aea17","Type":"ContainerStarted","Data":"58586b4ad392a0d6c2d88bd07fad9ca0d2aa701712af18ece4dcade426649d11"} Oct 30 19:52:08.686491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:08.686430 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" podStartSLOduration=2.686412081 podStartE2EDuration="2.686412081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:08.681435921 +0000 UTC m=+1288.749271489" watchObservedRunningTime="2026-10-30 19:52:08.686412081 +0000 UTC m=+1288.754247645" Oct 30 19:52:08.830066 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:08.830028 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:09.667927 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:09.667894 5733 generic.go:356] "Generic (PLEG): container finished" podID="e0d9f3fd-024d-4e24-97fa-cea7523aea17" containerID="58586b4ad392a0d6c2d88bd07fad9ca0d2aa701712af18ece4dcade426649d11" exitCode=0 Oct 30 19:52:09.668362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:09.667982 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" event={"ID":"e0d9f3fd-024d-4e24-97fa-cea7523aea17","Type":"ContainerDied","Data":"58586b4ad392a0d6c2d88bd07fad9ca0d2aa701712af18ece4dcade426649d11"} Oct 30 19:52:10.038537 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.038442 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.360979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.360944 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.826169 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.826128 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:10.917278 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.917247 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\") pod \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " Oct 30 19:52:10.917458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.917399 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\") pod \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " Oct 30 19:52:10.917610 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.917594 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\" (UniqueName: \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\") pod \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\" (UID: \"e0d9f3fd-024d-4e24-97fa-cea7523aea17\") " Oct 30 19:52:10.917723 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.917696 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca" pod "e0d9f3fd-024d-4e24-97fa-cea7523aea17" (UID: "e0d9f3fd-024d-4e24-97fa-cea7523aea17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:10.918027 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.918007 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:10.919658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.919635 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca" pod "e0d9f3fd-024d-4e24-97fa-cea7523aea17" (UID: "e0d9f3fd-024d-4e24-97fa-cea7523aea17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:10.919769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:10.919750 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq" pod "e0d9f3fd-024d-4e24-97fa-cea7523aea17" (UID: "e0d9f3fd-024d-4e24-97fa-cea7523aea17"). InnerVolumeSpecName "kube-api-access-2cctq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:11.019297 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.019250 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0d9f3fd-024d-4e24-97fa-cea7523aea17-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:11.019297 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.019282 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cctq\" (UniqueName: \"kubernetes.io/projected/e0d9f3fd-024d-4e24-97fa-cea7523aea17-kube-api-access-2cctq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:11.272637 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.272544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:11.684034 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.684002 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" Oct 30 19:52:11.684290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.684042 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l" event={"ID":"e0d9f3fd-024d-4e24-97fa-cea7523aea17","Type":"ContainerDied","Data":"a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f"} Oct 30 19:52:11.684290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:11.684115 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99d50eb153ec35c855b27332d24d114f65aaff13875334b849ef8cb1301bf4f" Oct 30 19:52:12.529167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:12.529122 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:13.778498 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:13.778457 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.704625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:14.704586 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:15.723190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:15.723162 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.997641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:16.997607 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:17.325857 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:17.325713 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:18.753008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:18.752932 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.156123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:20.156089 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.449619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:21.449579 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.485570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.485528 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.702093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.698849 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd"] Oct 30 19:52:22.702093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.700233 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e0d9f3fd-024d-4e24-97fa-cea7523aea17" containerName="revoke-keys" Oct 30 19:52:22.702093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.700257 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d9f3fd-024d-4e24-97fa-cea7523aea17" containerName="revoke-keys" Oct 30 19:52:22.702093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.700573 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d9f3fd-024d-4e24-97fa-cea7523aea17" containerName="revoke-keys" Oct 30 19:52:22.707407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.707274 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.712258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.712230 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd"] Oct 30 19:52:22.712986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.712962 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:22.714797 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.714776 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:52:22.823439 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.823348 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.823591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.823506 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.823681 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.823665 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfcb2\" (UniqueName: \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.924892 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.924858 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfcb2\" (UniqueName: \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.925091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.924935 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.925091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.925048 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.925735 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.925710 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.927401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.927375 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:22.938567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:22.938543 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfcb2\" (UniqueName: \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\") pod \"maas-api-revoke-sub-0119ee5a-zwptd\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:23.028598 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.028566 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:23.333399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.333370 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd"] Oct 30 19:52:23.333953 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:23.333927 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7d4b23_c74f_4f3c_8ad3_7eb9f6aeb096.slice/crio-4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363 WatchSource:0}: Error finding container 4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363: Status 404 returned error can't find the container with id 4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363 Oct 30 19:52:23.596258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.596213 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:23.884641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.884555 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" event={"ID":"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096","Type":"ContainerStarted","Data":"b8dc644b239af082bd0cf986ee45b20e8f0d8119954d345e1fc47d1cc2b24813"} Oct 30 19:52:23.884641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.884622 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" event={"ID":"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096","Type":"ContainerStarted","Data":"4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363"} Oct 30 19:52:23.909427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:23.909378 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" podStartSLOduration=1.909364157 podStartE2EDuration="1.909364157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:23.905415804 +0000 UTC m=+1303.973251376" watchObservedRunningTime="2026-10-30 19:52:23.909364157 +0000 UTC m=+1303.977199725" Oct 30 19:52:24.537353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:24.537310 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:24.894003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:24.893962 5733 generic.go:356] "Generic (PLEG): container finished" podID="7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" containerID="b8dc644b239af082bd0cf986ee45b20e8f0d8119954d345e1fc47d1cc2b24813" exitCode=0 Oct 30 19:52:24.894424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:24.894041 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" event={"ID":"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096","Type":"ContainerDied","Data":"b8dc644b239af082bd0cf986ee45b20e8f0d8119954d345e1fc47d1cc2b24813"} Oct 30 19:52:25.729183 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.729153 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 19:52:25.733380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.733354 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755606 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755639 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb44z\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755660 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755738 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755944 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755919 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.755997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.755980 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.756082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.756041 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.756126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.756114 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.756160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.756134 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.792651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.792613 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860600 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860659 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb44z\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860689 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860753 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860899 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.860946 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.861012 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.861075 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.861101 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.865964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.865554 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.866713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.866641 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.866713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.866656 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.866936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.866911 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.868869 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.867305 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.870097 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.869680 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.875703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.875680 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.888662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.888640 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:25.898280 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.898253 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.900525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:25.900499 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb44z\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:26.051272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.050514 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:26.086455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.086430 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:26.168596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.168555 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\") pod \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " Oct 30 19:52:26.168780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.168638 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\") pod \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " Oct 30 19:52:26.168780 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.168738 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\" (UniqueName: \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\") pod \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\" (UID: \"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096\") " Oct 30 19:52:26.169172 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.169131 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca" pod "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" (UID: "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:26.169480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.169457 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:26.170927 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.170898 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2" pod "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" (UID: "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096"). InnerVolumeSpecName "kube-api-access-zfcb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:26.171436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.171417 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca" pod "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" (UID: "7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:26.270706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.270669 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:26.270706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.270699 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfcb2\" (UniqueName: \"kubernetes.io/projected/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096-kube-api-access-zfcb2\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:26.346544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.346513 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 19:52:26.347334 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:26.347308 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd45e19_244b_44e1_87c0_d18e32f1de37.slice/crio-79a94be5d59d2475233d5210d0fe82c0a18bc9f8fb5e02145bd6241557cfc5a9 WatchSource:0}: Error finding container 79a94be5d59d2475233d5210d0fe82c0a18bc9f8fb5e02145bd6241557cfc5a9: Status 404 returned error can't find the container with id 79a94be5d59d2475233d5210d0fe82c0a18bc9f8fb5e02145bd6241557cfc5a9 Oct 30 19:52:26.350345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.350307 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:26.350530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.350378 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:26.350530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.350407 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:26.920078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.914406 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" event={"ID":"bdd45e19-244b-44e1-87c0-d18e32f1de37","Type":"ContainerStarted","Data":"397260cd646b5afb2919babcb13d1c99fbc636b9772d7a26649b696f99f20f7d"} Oct 30 19:52:26.920078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.914480 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" event={"ID":"bdd45e19-244b-44e1-87c0-d18e32f1de37","Type":"ContainerStarted","Data":"79a94be5d59d2475233d5210d0fe82c0a18bc9f8fb5e02145bd6241557cfc5a9"} Oct 30 19:52:26.939086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.937921 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" event={"ID":"7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096","Type":"ContainerDied","Data":"4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363"} Oct 30 19:52:26.939086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.937986 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afb7bddb3a543ebe446967fa2f1583a69b20b5de9080061c92f117bf8269363" Oct 30 19:52:26.939086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.938088 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd" Oct 30 19:52:26.957011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:26.956951 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podStartSLOduration=1.956932894 podStartE2EDuration="1.956932894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:26.954449682 +0000 UTC m=+1307.022285255" watchObservedRunningTime="2026-10-30 19:52:26.956932894 +0000 UTC m=+1307.024768453" Oct 30 19:52:27.052455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:27.052423 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:27.053854 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:27.053819 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:27.053991 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:27.053877 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:27.157707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:27.157608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:28.052016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:28.051980 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:28.052458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:28.052042 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:29.052447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:29.052414 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:29.052829 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:29.052484 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:29.184127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:29.184046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:29.377961 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:52:29.377926 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:52:30.052917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:30.052886 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:30.053331 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:30.052954 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:30.935090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:30.935030 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:31.052146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.052069 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:31.052343 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.052174 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:31.186263 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.186190 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 19:52:31.186874 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.186854 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" containerName="revoke-keys" Oct 30 19:52:31.186918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.186877 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" containerName="revoke-keys" Oct 30 19:52:31.187036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.187026 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" containerName="revoke-keys" Oct 30 19:52:31.190417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.190399 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.199770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.199745 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"distinct-w4-3f57c5-kserve-self-signed-certs\"" Oct 30 19:52:31.201819 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.201114 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 19:52:31.266100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.266026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362482 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362562 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362596 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362656 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362819 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f24t6\" (UniqueName: \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.363472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.362896 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463570 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f24t6\" (UniqueName: \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463644 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463699 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463760 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463792 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.463898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.463848 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.464374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.464350 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.464590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.464570 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.464949 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.464926 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.468234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.468209 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.468453 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.468431 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.478169 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.478144 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f24t6\" (UniqueName: \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\") pod \"distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.504205 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.504183 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:31.846608 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.846267 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 19:52:31.849446 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:31.849422 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64f65e7a_479c_44cd_9f40_5aea55deb3e1.slice/crio-8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3 WatchSource:0}: Error finding container 8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3: Status 404 returned error can't find the container with id 8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3 Oct 30 19:52:31.981302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.981264 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" event={"ID":"64f65e7a-479c-44cd-9f40-5aea55deb3e1","Type":"ContainerStarted","Data":"fbf88ad62aa4b42357dffdb6435bcd91149b05f865c57bb788f3b4b15f37aabc"} Oct 30 19:52:31.981302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:31.981307 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" event={"ID":"64f65e7a-479c-44cd-9f40-5aea55deb3e1","Type":"ContainerStarted","Data":"8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3"} Oct 30 19:52:32.004298 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:32.004247 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podStartSLOduration=1.004234308 podStartE2EDuration="1.004234308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:32.003604135 +0000 UTC m=+1312.071439739" watchObservedRunningTime="2026-10-30 19:52:32.004234308 +0000 UTC m=+1312.072069876" Oct 30 19:52:32.052706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:32.052663 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 30 19:52:32.052882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:32.052721 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 30 19:52:32.504443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:32.504412 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:33.058249 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.058216 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:33.058720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.058698 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:33.059693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.059667 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 19:52:33.796074 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.796002 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:33.802291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.802254 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 19:52:33.807638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.807614 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.839319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.839287 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"distinct-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 19:52:33.843190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.843154 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 19:52:33.898710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.898677 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.898710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.898719 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.898973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.898853 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmp96\" (UniqueName: \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.898973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.898904 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.898973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.898947 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:33.899150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:33.899020 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000286 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmp96\" (UniqueName: \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000353 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000392 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000451 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000517 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.000602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000542 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.001006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.000983 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.001042 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.001023 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.001567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.001305 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.003839 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.003817 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.004672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.004650 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.026031 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.026001 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmp96\" (UniqueName: \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\") pod \"distinct-w2-d5e3e5-kserve-c67646f86-96xsp\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.065332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.065173 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.131526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.131479 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:34.591766 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:34.591729 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 19:52:35.011886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:35.011848 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" event={"ID":"4e877fbf-b484-4cfc-93a9-b2e813a364b2","Type":"ContainerStarted","Data":"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c"} Oct 30 19:52:35.012357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:35.011904 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" event={"ID":"4e877fbf-b484-4cfc-93a9-b2e813a364b2","Type":"ContainerStarted","Data":"63f55dd8c84c08f4b13463ec4102929989b2a8ea54d75a55c083f12880d76003"} Oct 30 19:52:35.061967 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:35.061891 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podStartSLOduration=2.061871753 podStartE2EDuration="2.061871753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:35.060439172 +0000 UTC m=+1315.128274752" watchObservedRunningTime="2026-10-30 19:52:35.061871753 +0000 UTC m=+1315.129707322" Oct 30 19:52:35.132722 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:35.132682 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:37.147634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:37.147598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.855513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:38.855473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:39.953361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:39.953315 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.181436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.181403 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 19:52:40.186843 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.186814 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.201621 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.201587 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 19:52:40.205421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.205359 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"kube-root-ca.crt\"" Oct 30 19:52:40.205531 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.205514 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"default-dockercfg-2t28t\"" Oct 30 19:52:40.205595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.205582 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"distinct-2-w5-278ffa-kserve-self-signed-certs\"" Oct 30 19:52:40.207418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.207398 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"openshift-service-ca.crt\"" Oct 30 19:52:40.261290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261253 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rn58\" (UniqueName: \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.261487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261313 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.261487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261409 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.261627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261510 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.261627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261615 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.261759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.261740 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.342258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.342225 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.374832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.374791 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.374890 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rn58\" (UniqueName: \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.374926 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.374946 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.375022 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.375121 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.375898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.375868 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.376018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.375974 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.376018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.376001 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.378201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.378176 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.382294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.382268 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.392489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.392460 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rn58\" (UniqueName: \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\") pod \"distinct-2-w5-278ffa-kserve-c69ccf475-jppkh\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.506524 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.506434 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:40.820165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.820131 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 19:52:40.822579 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:40.822541 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd142cf77_146e_4719_8e79_001e99aead3c.slice/crio-ed1ac0a33338e4eaae0454dfcacc680cc46005dceba36501c51b79ab2efe304c WatchSource:0}: Error finding container ed1ac0a33338e4eaae0454dfcacc680cc46005dceba36501c51b79ab2efe304c: Status 404 returned error can't find the container with id ed1ac0a33338e4eaae0454dfcacc680cc46005dceba36501c51b79ab2efe304c Oct 30 19:52:40.870221 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.870193 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 19:52:40.875388 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.875362 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.884938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.884858 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 19:52:40.984188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984151 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984274 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984311 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984368 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984460 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984540 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984558 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8lnz\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984685 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:40.984750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:40.984725 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.059027 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.058986 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" event={"ID":"d142cf77-146e-4719-8e79-001e99aead3c","Type":"ContainerStarted","Data":"ab194f18663215340d74d8ca2e3e3a0acbce4171615837b6a63db0b571421623"} Oct 30 19:52:41.059219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.059039 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" event={"ID":"d142cf77-146e-4719-8e79-001e99aead3c","Type":"ContainerStarted","Data":"ed1ac0a33338e4eaae0454dfcacc680cc46005dceba36501c51b79ab2efe304c"} Oct 30 19:52:41.085965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.085890 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.085965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.085926 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.085976 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086010 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086030 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086053 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086112 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086162 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086509 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086184 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8lnz\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086520 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086552 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086670 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086764 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086670 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.086892 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.086866 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.088519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.088491 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.088619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.088605 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.094326 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.094283 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podStartSLOduration=1.0942706229999999 podStartE2EDuration="1.094270623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:41.090735451 +0000 UTC m=+1321.158571021" watchObservedRunningTime="2026-10-30 19:52:41.094270623 +0000 UTC m=+1321.162106191" Oct 30 19:52:41.102627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.102608 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.103102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.103084 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8lnz\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.202156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.202128 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:41.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.231900 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.505335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.505299 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:41.506757 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.506728 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:41.526820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.526791 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:41.561134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.561108 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 19:52:41.561252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.561106 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:41.561252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.561209 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:41.561375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:41.561281 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:52:42.067554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.067509 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" event={"ID":"4b1744f9-9ed1-4e7b-a359-8a1499160126","Type":"ContainerStarted","Data":"432546abde812b81a38c559ac44aedba642da56220811edccaeeb6b39c49d08d"} Oct 30 19:52:42.068035 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.067572 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" event={"ID":"4b1744f9-9ed1-4e7b-a359-8a1499160126","Type":"ContainerStarted","Data":"08b17320cec58d4aa6a4a8cf298871bdbeff031ff70594c09521eaa223e143ec"} Oct 30 19:52:42.079418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.079395 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:52:42.118710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.118663 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podStartSLOduration=2.118650015 podStartE2EDuration="2.118650015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:42.115090402 +0000 UTC m=+1322.182925971" watchObservedRunningTime="2026-10-30 19:52:42.118650015 +0000 UTC m=+1322.186485653" Oct 30 19:52:42.202369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.202299 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:42.203519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.203494 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:42.203620 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.203541 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:42.517819 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:42.517568 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:43.202877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:43.202838 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:43.203374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:43.202893 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:43.569134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:43.569034 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:44.134547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:44.134512 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:44.159278 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:44.159246 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:44.204585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:44.204548 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:44.205230 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:44.205197 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:45.020583 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:45.020529 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:45.207404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:45.206931 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:45.207404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:45.207003 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:45.294665 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:45.294570 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:52:46.202947 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:46.202901 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:46.203208 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:46.202971 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:46.342490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:46.342452 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.202684 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:47.202644 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:47.202963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:47.202929 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:47.592139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:47.589978 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:48.202546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:48.202498 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 30 19:52:48.202765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:48.202557 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 30 19:52:48.709124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:48.705305 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:49.208029 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:49.207994 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:49.208451 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:49.208425 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:49.209105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:49.209085 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 19:52:50.446408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:50.446365 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:50.507556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:50.507524 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:50.524053 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:50.524026 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:51.177449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:51.177415 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:51.356840 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:51.356807 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:52:52.111405 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:52.111371 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:53.036841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:53.036803 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.248925 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.248882 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.487354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.487322 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 19:52:54.492699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.492670 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.499922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.499854 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"distinct-w6-5f933e-kserve-self-signed-certs\"" Oct 30 19:52:54.502558 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.502526 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 19:52:54.584887 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.584847 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.642039 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642001 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.642257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642142 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.642257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642251 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.642398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642282 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.642458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642409 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-594d9\" (UniqueName: \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.642510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.642460 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.743874 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.743816 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.744103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.743922 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.744103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.744027 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.744103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.744088 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.744295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.744160 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-594d9\" (UniqueName: \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.744295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.744218 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.745158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.745126 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.745330 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.745308 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.745524 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.745490 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.747987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.747961 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.749198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.749167 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.771725 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.771648 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-594d9\" (UniqueName: \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\") pod \"distinct-w6-5f933e-kserve-594b74cb-9fzpf\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:54.810516 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:54.810486 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:55.114637 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.114605 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 19:52:55.116629 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:55.116594 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1c5d55c_b95f_4c74_9bbb_4d660b088cfa.slice/crio-e628c22770f21328ed59a671ca928aadfc1ed4acb6c412fca70e069cce2e6723 WatchSource:0}: Error finding container e628c22770f21328ed59a671ca928aadfc1ed4acb6c412fca70e069cce2e6723: Status 404 returned error can't find the container with id e628c22770f21328ed59a671ca928aadfc1ed4acb6c412fca70e069cce2e6723 Oct 30 19:52:55.387650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.384895 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" event={"ID":"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa","Type":"ContainerStarted","Data":"857f0b4926b80f58c968766f1dbe7be8a3aa87d24ebba0da98d85dd47ae72141"} Oct 30 19:52:55.387650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.384946 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" event={"ID":"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa","Type":"ContainerStarted","Data":"e628c22770f21328ed59a671ca928aadfc1ed4acb6c412fca70e069cce2e6723"} Oct 30 19:52:55.468217 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.449199 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.468217 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.451300 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podStartSLOduration=1.451280136 podStartE2EDuration="1.451280136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:55.444780618 +0000 UTC m=+1335.512616187" watchObservedRunningTime="2026-10-30 19:52:55.451280136 +0000 UTC m=+1335.519115707" Oct 30 19:52:55.814919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:55.814774 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:52:56.427049 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:56.427005 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:56.698568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:56.698473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.065570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.065520 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.290966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.290907 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.319568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.319494 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 19:52:58.323468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.323451 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.331719 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.331459 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-w3-8a8207-kserve-self-signed-certs\"" Oct 30 19:52:58.332365 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.332341 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"openshift-service-ca.crt\"" Oct 30 19:52:58.332510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.332343 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"default-dockercfg-lb6sq\"" Oct 30 19:52:58.332510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.332410 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"kube-root-ca.crt\"" Oct 30 19:52:58.340274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.340249 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 19:52:58.433507 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433456 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.433698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433535 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.433698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433571 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.433824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433754 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.433824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433813 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.433937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.433883 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67sl\" (UniqueName: \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.490119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.490048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:58.535644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.535606 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.535857 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.535731 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.535857 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.535777 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.535988 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.535970 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.536047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.536035 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.536135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.536121 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67sl\" (UniqueName: \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.536974 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.536951 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.537631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.537602 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.537923 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.537900 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.539468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.539438 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.540142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.540120 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.553341 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.553315 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67sl\" (UniqueName: \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\") pod \"distinct-w3-8a8207-kserve-68f4987c5d-zbc97\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.637594 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.637562 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:52:58.950337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:58.950303 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 19:52:58.950810 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:52:58.950776 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11fec734_033b_44bb_a7a4_5a3f81ecddeb.slice/crio-fb69c8a5ea37d0cc3bc625279a43292caf9e49a46c15971ddffc03706779fdee WatchSource:0}: Error finding container fb69c8a5ea37d0cc3bc625279a43292caf9e49a46c15971ddffc03706779fdee: Status 404 returned error can't find the container with id fb69c8a5ea37d0cc3bc625279a43292caf9e49a46c15971ddffc03706779fdee Oct 30 19:52:59.169977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:59.169934 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.377917 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:52:59.377881 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:52:59.419672 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:59.419630 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" event={"ID":"11fec734-033b-44bb-a7a4-5a3f81ecddeb","Type":"ContainerStarted","Data":"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d"} Oct 30 19:52:59.419867 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:59.419688 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" event={"ID":"11fec734-033b-44bb-a7a4-5a3f81ecddeb","Type":"ContainerStarted","Data":"fb69c8a5ea37d0cc3bc625279a43292caf9e49a46c15971ddffc03706779fdee"} Oct 30 19:52:59.442248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:59.442194 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podStartSLOduration=1.442180341 podStartE2EDuration="1.442180341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:59.440926134 +0000 UTC m=+1339.508761705" watchObservedRunningTime="2026-10-30 19:52:59.442180341 +0000 UTC m=+1339.510015908" Oct 30 19:52:59.638468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:52:59.638385 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:53:00.221742 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:00.221685 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:01.028545 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:01.028509 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.386001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:02.385942 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:03.292079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:03.292023 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.326893 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:04.326796 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:04.811192 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:04.811152 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:53:04.823777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:04.823751 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:53:05.149173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:05.149126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.323385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:05.323354 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.475310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:05.475232 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:53:06.104488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:06.104448 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:07.179976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:07.179941 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.639465 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:08.639429 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:53:09.053135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:09.051728 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.240084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:09.239760 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:53:09.530646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:09.530611 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:53:10.210189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.210108 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 19:53:10.213352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.213299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.218305 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.218279 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.246183 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.246147 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 19:53:10.261542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.261505 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.262228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.262203 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 19:53:10.268625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.268602 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 19:53:10.295332 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295302 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295341 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdcxc\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.295518 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295456 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.295518 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295473 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295575 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.295605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295591 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295608 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295624 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qng\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295643 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295712 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.295808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295756 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295873 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295852 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.295922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295910 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.295964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.295931 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.296018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.296001 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.296085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.296040 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.296144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.296095 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.296199 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.296143 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397243 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397277 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397331 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397347 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397375 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397522 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397393 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qng\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397528 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397572 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397602 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397630 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397700 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397704 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.397759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397734 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397783 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397801 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397804 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397853 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397892 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397934 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.397999 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398035 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdcxc\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398099 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398130 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398156 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398217 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398429 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.398805 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398702 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.398994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.398970 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.400350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.400329 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.400475 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.400433 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.400568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.400551 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.400629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.400607 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.425487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.425458 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.426204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.426178 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdcxc\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.426610 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.426588 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qng\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.432706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.432680 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.540190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.540103 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:10.581240 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.581204 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:10.846294 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:10.846249 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fd6a8d_27a4_40c4_b7db_7d7e0fccd46e.slice/crio-6ae3cc859c616c5ba4bf280861482943073c800db7f20459e294d4ce1eb34788 WatchSource:0}: Error finding container 6ae3cc859c616c5ba4bf280861482943073c800db7f20459e294d4ce1eb34788: Status 404 returned error can't find the container with id 6ae3cc859c616c5ba4bf280861482943073c800db7f20459e294d4ce1eb34788 Oct 30 19:53:10.846771 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.846747 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 19:53:10.849382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.849337 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:10.849479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.849448 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:10.849519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.849491 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:10.898611 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.898583 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 19:53:10.899555 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:10.899530 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5652aad_c74e_4732_8bd7_1d814d64be92.slice/crio-9b014d8a4b254591ac921c5efb64bec7d105a6e8799ebffe42ea0435e4dea529 WatchSource:0}: Error finding container 9b014d8a4b254591ac921c5efb64bec7d105a6e8799ebffe42ea0435e4dea529: Status 404 returned error can't find the container with id 9b014d8a4b254591ac921c5efb64bec7d105a6e8799ebffe42ea0435e4dea529 Oct 30 19:53:10.902106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.902071 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:10.902171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.902146 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:10.902215 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:10.902176 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:11.542083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.541122 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" event={"ID":"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e","Type":"ContainerStarted","Data":"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212"} Oct 30 19:53:11.542083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.541194 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" event={"ID":"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e","Type":"ContainerStarted","Data":"6ae3cc859c616c5ba4bf280861482943073c800db7f20459e294d4ce1eb34788"} Oct 30 19:53:11.548076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.544961 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" event={"ID":"f5652aad-c74e-4732-8bd7-1d814d64be92","Type":"ContainerStarted","Data":"4c4761934862cdc7a3f6b2cc0d1c5b9799955b1bb161babf36a63c9a37329ce6"} Oct 30 19:53:11.548076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.545026 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" event={"ID":"f5652aad-c74e-4732-8bd7-1d814d64be92","Type":"ContainerStarted","Data":"9b014d8a4b254591ac921c5efb64bec7d105a6e8799ebffe42ea0435e4dea529"} Oct 30 19:53:11.548076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.546044 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:11.551389 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.551353 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:11.551554 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.551427 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:11.563534 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.563258 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.578917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.578858 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podStartSLOduration=1.578844405 podStartE2EDuration="1.578844405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.577139594 +0000 UTC m=+1351.644975163" watchObservedRunningTime="2026-10-30 19:53:11.578844405 +0000 UTC m=+1351.646679972" Oct 30 19:53:11.582357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.582331 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:11.584047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.584022 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:11.584208 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.584089 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:11.648081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:11.632496 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podStartSLOduration=1.632476855 podStartE2EDuration="1.632476855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.626667718 +0000 UTC m=+1351.694503291" watchObservedRunningTime="2026-10-30 19:53:11.632476855 +0000 UTC m=+1351.700312422" Oct 30 19:53:12.548094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:12.545998 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:12.548094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:12.546076 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:12.588104 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:12.584551 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:12.588104 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:12.584613 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:12.592128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:12.588708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.536544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:13.536485 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.544080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:13.540902 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:13.544080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:13.540972 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:13.604519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:13.604483 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:13.605167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:13.605124 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:14.547080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:14.544168 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:14.547080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:14.544238 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:14.582445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:14.582401 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:14.582618 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:14.582463 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:14.657083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:14.638120 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:15.553081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:15.545568 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:15.553081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:15.545638 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:15.588749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:15.587854 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:15.588749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:15.587921 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:16.566406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:16.565247 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:16.566406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:16.565315 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:16.586559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:16.586526 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:16.587848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:16.587810 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:17.286702 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.286636 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.456671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.456631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.540556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.540457 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:53:17.540728 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.540545 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 19:53:17.582578 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.582538 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:53:17.583002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.582607 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 19:53:17.914932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:17.914891 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:18.546699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.546671 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:18.547165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.547143 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:18.548035 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.548013 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 19:53:18.586865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.586839 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:18.587248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.587166 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:18.587868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:18.587840 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 19:53:19.005372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:19.003291 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:20.212471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:20.212431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:21.076813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:21.076733 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.114086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:22.112982 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:23.233012 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:23.232945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:23.439762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:23.439680 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.504623 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:24.504589 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:25.646603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:25.646562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.093744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.093719 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 19:53:26.099178 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.099155 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.121913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.121882 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzs5\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.122179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.121976 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.122179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122013 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.122179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122069 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.122179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122096 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.123204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122223 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.123204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122247 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.123204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122284 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.123204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.122366 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.126793 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.126764 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 19:53:26.224067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224028 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224098 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224139 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224159 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224260 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224284 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224322 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224404 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224448 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzs5\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224574 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.224893 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224872 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.225722 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.224987 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.225828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.225169 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.225828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.225657 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.226970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.226941 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.227181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.227156 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.244753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.244719 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.245370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.245341 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzs5\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.414451 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.414355 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:26.732390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.732321 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 19:53:26.735865 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:26.735838 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb535075b_1639_47bb_b428_7c5c0eb5d4b2.slice/crio-bdc914c13504880180034edbcca0cc9929926bb2e194b7e03dadc749de24380e WatchSource:0}: Error finding container bdc914c13504880180034edbcca0cc9929926bb2e194b7e03dadc749de24380e: Status 404 returned error can't find the container with id bdc914c13504880180034edbcca0cc9929926bb2e194b7e03dadc749de24380e Oct 30 19:53:26.739147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.739103 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:26.739244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.739192 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:26.739244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.739233 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:26.871079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.870010 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" event={"ID":"b535075b-1639-47bb-b428-7c5c0eb5d4b2","Type":"ContainerStarted","Data":"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635"} Oct 30 19:53:26.871079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.870110 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" event={"ID":"b535075b-1639-47bb-b428-7c5c0eb5d4b2","Type":"ContainerStarted","Data":"bdc914c13504880180034edbcca0cc9929926bb2e194b7e03dadc749de24380e"} Oct 30 19:53:26.899232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.899199 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:26.927899 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:26.927838 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podStartSLOduration=0.927818747 podStartE2EDuration="927.818747ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:26.925201399 +0000 UTC m=+1366.993036993" watchObservedRunningTime="2026-10-30 19:53:26.927818747 +0000 UTC m=+1366.995654319" Oct 30 19:53:27.415429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.415387 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:27.416914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.416879 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:27.417070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.416939 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:27.779557 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.779478 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv"] Oct 30 19:53:27.783644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.783624 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.814321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.814294 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:27.823303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.823276 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:53:27.854169 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.854127 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv"] Oct 30 19:53:27.861975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.861943 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.862313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.862280 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkwh4\" (UniqueName: \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.862454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.862434 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.963408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.963371 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.963629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.963607 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.963841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.963815 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkwh4\" (UniqueName: \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.964507 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.964479 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.966190 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.966164 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:27.988222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:27.988199 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkwh4\" (UniqueName: \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\") pod \"maas-api-revoke-sub-a9b928fd-jgzgv\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:28.075287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.075210 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:28.097369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.097339 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:28.431080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.428710 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:28.431080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.428777 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:28.481613 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:28.481583 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8608f56_0718_4176_8ff2_52cca3bbabb7.slice/crio-86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67 WatchSource:0}: Error finding container 86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67: Status 404 returned error can't find the container with id 86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67 Oct 30 19:53:28.482130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.482107 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv"] Oct 30 19:53:28.893559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:28.893526 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" event={"ID":"e8608f56-0718-4176-8ff2-52cca3bbabb7","Type":"ContainerStarted","Data":"86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67"} Oct 30 19:53:29.378481 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:53:29.378094 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:53:29.404553 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:29.404474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.432965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:29.432496 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:29.432965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:29.432560 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:29.901862 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:29.901826 5733 generic.go:356] "Generic (PLEG): container finished" podID="e8608f56-0718-4176-8ff2-52cca3bbabb7" containerID="33e58c6ce46c2f9d9706f08d23b20cd4d59a426267d72b3030a03c0bb5e68a28" exitCode=0 Oct 30 19:53:29.902262 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:29.901912 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" event={"ID":"e8608f56-0718-4176-8ff2-52cca3bbabb7","Type":"ContainerDied","Data":"33e58c6ce46c2f9d9706f08d23b20cd4d59a426267d72b3030a03c0bb5e68a28"} Oct 30 19:53:30.428082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:30.427423 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:30.428082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:30.427481 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:30.685430 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:30.685318 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.163875 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.163848 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:31.239206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.232531 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\") pod \"e8608f56-0718-4176-8ff2-52cca3bbabb7\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " Oct 30 19:53:31.239206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.232789 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\" (UniqueName: \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\") pod \"e8608f56-0718-4176-8ff2-52cca3bbabb7\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " Oct 30 19:53:31.239206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.232831 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\") pod \"e8608f56-0718-4176-8ff2-52cca3bbabb7\" (UID: \"e8608f56-0718-4176-8ff2-52cca3bbabb7\") " Oct 30 19:53:31.239206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.233781 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca" pod "e8608f56-0718-4176-8ff2-52cca3bbabb7" (UID: "e8608f56-0718-4176-8ff2-52cca3bbabb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:31.245081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.239774 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca" pod "e8608f56-0718-4176-8ff2-52cca3bbabb7" (UID: "e8608f56-0718-4176-8ff2-52cca3bbabb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:31.247314 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.247249 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4" pod "e8608f56-0718-4176-8ff2-52cca3bbabb7" (UID: "e8608f56-0718-4176-8ff2-52cca3bbabb7"). InnerVolumeSpecName "kube-api-access-wkwh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:31.342219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.342175 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkwh4\" (UniqueName: \"kubernetes.io/projected/e8608f56-0718-4176-8ff2-52cca3bbabb7-kube-api-access-wkwh4\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.342219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.342213 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.342219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.342228 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8608f56-0718-4176-8ff2-52cca3bbabb7-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.421083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.420408 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:31.421083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.420469 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:31.761372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.761329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:31.945137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.945089 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" event={"ID":"e8608f56-0718-4176-8ff2-52cca3bbabb7","Type":"ContainerDied","Data":"86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67"} Oct 30 19:53:31.945348 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.945194 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f552e3a9506f4bf3ce783f3834f436afa4708ddb5cea816b2b19bf333e7b67" Oct 30 19:53:31.945348 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:31.945262 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv" Oct 30 19:53:32.415465 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:32.415426 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:32.416080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:32.416019 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:32.870894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:32.870839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.415488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:33.415455 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:53:33.416013 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:33.415521 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Oct 30 19:53:34.431900 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:34.431869 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:34.433989 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:34.433962 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:34.441382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:34.441346 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 19:53:34.731527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:34.728947 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.635119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.609102 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 19:53:35.648189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.610050 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e8608f56-0718-4176-8ff2-52cca3bbabb7" containerName="revoke-keys" Oct 30 19:53:35.648189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.638999 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8608f56-0718-4176-8ff2-52cca3bbabb7" containerName="revoke-keys" Oct 30 19:53:35.648189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.639240 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8608f56-0718-4176-8ff2-52cca3bbabb7" containerName="revoke-keys" Oct 30 19:53:35.678085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.671285 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.732091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.692097 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"default-dockercfg-fzr2x\"" Oct 30 19:53:35.757747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.700381 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 19:53:35.758002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.700528 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"kube-root-ca.crt\"" Oct 30 19:53:35.758141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.701211 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"openshift-service-ca.crt\"" Oct 30 19:53:35.783079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.701328 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"distinct-2-w1-a1d885-kserve-self-signed-certs\"" Oct 30 19:53:35.784249 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.718724 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.784709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.784677 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.849082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.803661 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.923119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.923005 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.924043 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.924017 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.924383 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.924364 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:35.924691 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:35.924668 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpjp\" (UniqueName: \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.041965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.041920 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.042274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.042245 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpjp\" (UniqueName: \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.042502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.042481 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.042681 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.042662 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.042823 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.042807 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.043169 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.043144 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.097968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.097934 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.098487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.098461 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.132938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.073254 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.215129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.213188 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpjp\" (UniqueName: \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.235082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.233474 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.255112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.248721 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\") pod \"distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.316110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.315179 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.366082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.351135 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.502279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.502177 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:53:36.541370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.541325 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:36.546856 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.546150 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.688153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.688093 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.731235 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.731121 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:36.783357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.782761 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.784700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.784606 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:36.809631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:36.809134 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:53:37.127241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:37.120765 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:53:37.337665 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:37.337636 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 19:53:37.339820 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:37.339790 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd550cab4_61b4_43c5_ae10_4eb9597cc7b0.slice/crio-38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70 WatchSource:0}: Error finding container 38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70: Status 404 returned error can't find the container with id 38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70 Oct 30 19:53:38.129275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.129233 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.174274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.174233 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" event={"ID":"d550cab4-61b4-43c5-ae10-4eb9597cc7b0","Type":"ContainerStarted","Data":"c6233051842bab00187e40b121026c0d086b1ad1bdcf91427bb44fc1054ae406"} Oct 30 19:53:38.174444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.174283 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" event={"ID":"d550cab4-61b4-43c5-ae10-4eb9597cc7b0","Type":"ContainerStarted","Data":"38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70"} Oct 30 19:53:38.208562 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.208513 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podStartSLOduration=3.208499826 podStartE2EDuration="3.208499826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:38.206660678 +0000 UTC m=+1378.274496248" watchObservedRunningTime="2026-10-30 19:53:38.208499826 +0000 UTC m=+1378.276335393" Oct 30 19:53:38.541960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.541873 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:38.849049 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.849013 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 19:53:38.853303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.853279 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.930366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.930329 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 19:53:38.938367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.938340 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-2-w3-8a8207-kserve-self-signed-certs\"" Oct 30 19:53:38.951333 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951300 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.951495 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951401 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn666\" (UniqueName: \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.951495 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951459 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.951635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951611 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.951760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951657 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:38.951760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:38.951745 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.055440 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.055533 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn666\" (UniqueName: \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.056265 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.058245 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.058621 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.058872 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.059139 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.059048 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.060085 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.062814 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.063765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.063418 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.084837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.079568 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:39.088609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.088575 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn666\" (UniqueName: \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\") pod \"distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.165758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.165677 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:39.505830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.505798 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 19:53:39.506035 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:39.506013 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3c61292_cd14_4230_80ac_2aaf2d55be7c.slice/crio-b4ca4f8964c48bcd24d93c0d7575361747264f0db0da49f23da72808a03f55c3 WatchSource:0}: Error finding container b4ca4f8964c48bcd24d93c0d7575361747264f0db0da49f23da72808a03f55c3: Status 404 returned error can't find the container with id b4ca4f8964c48bcd24d93c0d7575361747264f0db0da49f23da72808a03f55c3 Oct 30 19:53:39.852017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:39.851984 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.197607 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.197531 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" event={"ID":"f3c61292-cd14-4230-80ac-2aaf2d55be7c","Type":"ContainerStarted","Data":"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7"} Oct 30 19:53:40.197607 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.197571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" event={"ID":"f3c61292-cd14-4230-80ac-2aaf2d55be7c","Type":"ContainerStarted","Data":"b4ca4f8964c48bcd24d93c0d7575361747264f0db0da49f23da72808a03f55c3"} Oct 30 19:53:40.220052 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.220007 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podStartSLOduration=2.219995071 podStartE2EDuration="2.219995071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:40.218213324 +0000 UTC m=+1380.286048892" watchObservedRunningTime="2026-10-30 19:53:40.219995071 +0000 UTC m=+1380.287830692" Oct 30 19:53:40.536897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.536817 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 19:53:40.541493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.541472 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.567328 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.567295 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 19:53:40.574742 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.574712 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 19:53:40.579908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.579883 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.580741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.580711 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.580855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.580757 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.580933 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.580903 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.580976 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.581036 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b4hp\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.581128 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.581423 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.581464 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.581580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.581494 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.600207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.600176 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 19:53:40.682766 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.682719 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.682934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.682798 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b4hp\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.682934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.682842 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.682934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.682886 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.682934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.682923 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683201 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683217 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683312 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683233 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683318 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683344 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683423 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683396 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683459 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683436 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683471 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683495 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xlb\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683524 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683547 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683599 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683684 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683643 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.683715 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683686 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.683754 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.683710 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.684521 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.684494 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.684866 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.684840 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.685078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.685038 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.686427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.686400 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.686716 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.686699 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.697255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.697229 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b4hp\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.698341 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.698318 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.729409 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.729364 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.785547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785503 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785578 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785594 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785645 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785674 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785826 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785854 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785881 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xlb\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.785960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.785910 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.786242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.786041 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.786408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.786385 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.786443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.786409 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.786443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.786410 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.786518 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.786500 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.787996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.787895 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.788492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.788473 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.806026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.805995 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xlb\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.806831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.806802 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:40.857752 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.857721 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:40.906380 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:40.906338 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:41.166890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.166845 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 19:53:41.167073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.167010 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:41.173688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.173644 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.173785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.173736 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.173830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.173785 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.212513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.212483 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" event={"ID":"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0","Type":"ContainerStarted","Data":"052d1f9009159756835233253f626c82916ace65b505533654729362e62c1201"} Oct 30 19:53:41.221580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.221553 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 19:53:41.227383 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:41.225498 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db2cd8e_f002_40ae_9ca1_66041fdec5bd.slice/crio-4c591954c398f61cd32efdc1aaa47886b028a283c5f5760f2d1e8049601dde53 WatchSource:0}: Error finding container 4c591954c398f61cd32efdc1aaa47886b028a283c5f5760f2d1e8049601dde53: Status 404 returned error can't find the container with id 4c591954c398f61cd32efdc1aaa47886b028a283c5f5760f2d1e8049601dde53 Oct 30 19:53:41.228125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.228080 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.228224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.228173 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.228281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.228219 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:41.316027 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.315994 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 19:53:41.326091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.326070 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.343923 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.343881 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401691 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvht\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401767 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401835 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401875 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401937 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.401987 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.402048 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.402117 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.405129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.402151 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503260 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503331 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503370 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503437 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503482 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503538 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503587 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503727 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503633 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.503875 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.503765 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvht\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.504131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.504098 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.504386 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.504362 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.504455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.504380 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.504717 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.504699 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.504862 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.504843 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.507319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.507296 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.508602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.508560 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.520319 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.520291 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.524395 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.524367 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvht\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.678880 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.678787 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6"] Oct 30 19:53:41.685234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.685212 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.687420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.687156 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:41.692160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.692136 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:41.692296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.692241 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:53:41.701383 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.697382 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6"] Oct 30 19:53:41.709102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.706702 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.709102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.706806 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.709102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.707028 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rlk\" (UniqueName: \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.800928 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.800901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:41.842367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.809390 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rlk\" (UniqueName: \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.842367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.809501 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.842367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.809586 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.842367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.812370 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.846072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.846019 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rlk\" (UniqueName: \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:41.860437 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:41.860282 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f3744e-q49h6\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:42.004652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.004466 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:42.033538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.033489 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 19:53:42.060040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.059339 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:42.060040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.059424 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:42.060040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.059464 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:42.297148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.296257 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" event={"ID":"9db2cd8e-f002-40ae-9ca1-66041fdec5bd","Type":"ContainerStarted","Data":"ea53bfaf7fa074f187adf9ab5649a7b4786373a9fbeb6dc775ecb810ea2c2df3"} Oct 30 19:53:42.297148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.296332 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" event={"ID":"9db2cd8e-f002-40ae-9ca1-66041fdec5bd","Type":"ContainerStarted","Data":"4c591954c398f61cd32efdc1aaa47886b028a283c5f5760f2d1e8049601dde53"} Oct 30 19:53:42.301677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.301552 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" event={"ID":"b9558d5f-e925-43dc-90b4-928cd373c624","Type":"ContainerStarted","Data":"f1f15dca5e73b7c55271a553ff58fdc66d89066c623b6b4086ada73b2da21a60"} Oct 30 19:53:42.301677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.301618 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" event={"ID":"b9558d5f-e925-43dc-90b4-928cd373c624","Type":"ContainerStarted","Data":"ecb143c105945f3d38646672f5fb831637cd512e835b33a6fe1edbde532e65a2"} Oct 30 19:53:42.305432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.305370 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" event={"ID":"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0","Type":"ContainerStarted","Data":"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f"} Oct 30 19:53:42.486549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.486466 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podStartSLOduration=2.4864447849999998 podStartE2EDuration="2.486444785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.391489975 +0000 UTC m=+1382.459325544" watchObservedRunningTime="2026-10-30 19:53:42.486444785 +0000 UTC m=+1382.554280358" Oct 30 19:53:42.487416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.487353 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6"] Oct 30 19:53:42.499938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.499888 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podStartSLOduration=2.499874884 podStartE2EDuration="2.499874884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.490370145 +0000 UTC m=+1382.558205715" watchObservedRunningTime="2026-10-30 19:53:42.499874884 +0000 UTC m=+1382.567710452" Oct 30 19:53:42.590559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.590483 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podStartSLOduration=1.590452869 podStartE2EDuration="1.590452869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:42.578357154 +0000 UTC m=+1382.646192726" watchObservedRunningTime="2026-10-30 19:53:42.590452869 +0000 UTC m=+1382.658288437" Oct 30 19:53:42.687480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.687433 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:42.860663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.860629 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:42.878262 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.878219 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:42.878443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.878287 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:42.907134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.907106 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:42.908470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.908442 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:42.908613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:42.908489 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:43.121139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.117987 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:43.406096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.405973 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" event={"ID":"701faa90-8cba-4b4e-a189-ab1db587a0c0","Type":"ContainerStarted","Data":"3ffbd318996035f213088e8374430869cb054500e07a3056cc2da16ff99ecbe5"} Oct 30 19:53:43.406096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.406040 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" event={"ID":"701faa90-8cba-4b4e-a189-ab1db587a0c0","Type":"ContainerStarted","Data":"c2363f9025a0c81186d1e2857116a744570fd0d4d724f580a6422153118bcd2c"} Oct 30 19:53:43.470082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.469776 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" podStartSLOduration=2.469752096 podStartE2EDuration="2.469752096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:43.443572836 +0000 UTC m=+1383.511408407" watchObservedRunningTime="2026-10-30 19:53:43.469752096 +0000 UTC m=+1383.537587683" Oct 30 19:53:43.694147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.693516 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:43.694147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.693583 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:43.902482 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.902339 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:43.902822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.902775 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:43.908041 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.908011 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:43.908300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:43.908275 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:44.133835 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.133800 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.678965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.665413 5733 generic.go:356] "Generic (PLEG): container finished" podID="701faa90-8cba-4b4e-a189-ab1db587a0c0" containerID="3ffbd318996035f213088e8374430869cb054500e07a3056cc2da16ff99ecbe5" exitCode=0 Oct 30 19:53:44.678965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.667109 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" event={"ID":"701faa90-8cba-4b4e-a189-ab1db587a0c0","Type":"ContainerDied","Data":"3ffbd318996035f213088e8374430869cb054500e07a3056cc2da16ff99ecbe5"} Oct 30 19:53:44.698424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.698389 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:44.698669 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.698638 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:44.858506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.858470 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:44.858773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.858745 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:44.909227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.909190 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:44.909500 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:44.909470 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:45.210614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.210575 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:45.701233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.689500 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:45.701233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.689566 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:45.883082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.881821 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:45.883082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.882181 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:45.909764 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.909623 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:45.909764 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:45.909678 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:46.370848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.367623 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:46.653143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.600084 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:46.744244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.742917 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:46.744244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.742986 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:46.841105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.840795 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:46.854652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.854614 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\" (UniqueName: \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\") pod \"701faa90-8cba-4b4e-a189-ab1db587a0c0\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " Oct 30 19:53:46.854988 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.854963 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\") pod \"701faa90-8cba-4b4e-a189-ab1db587a0c0\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " Oct 30 19:53:46.855382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.855359 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\") pod \"701faa90-8cba-4b4e-a189-ab1db587a0c0\" (UID: \"701faa90-8cba-4b4e-a189-ab1db587a0c0\") " Oct 30 19:53:46.873105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.873051 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca" pod "701faa90-8cba-4b4e-a189-ab1db587a0c0" (UID: "701faa90-8cba-4b4e-a189-ab1db587a0c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:46.898757 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.898723 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:46.899000 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.898969 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:46.970789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.970442 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" event={"ID":"701faa90-8cba-4b4e-a189-ab1db587a0c0","Type":"ContainerDied","Data":"c2363f9025a0c81186d1e2857116a744570fd0d4d724f580a6422153118bcd2c"} Oct 30 19:53:46.970789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.970511 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2363f9025a0c81186d1e2857116a744570fd0d4d724f580a6422153118bcd2c" Oct 30 19:53:46.970789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:46.970584 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6" Oct 30 19:53:47.000184 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.000149 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:47.021080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.014314 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk" pod "701faa90-8cba-4b4e-a189-ab1db587a0c0" (UID: "701faa90-8cba-4b4e-a189-ab1db587a0c0"). InnerVolumeSpecName "kube-api-access-g5rlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:47.021080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.020212 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca" pod "701faa90-8cba-4b4e-a189-ab1db587a0c0" (UID: "701faa90-8cba-4b4e-a189-ab1db587a0c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:47.073458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.073421 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:47.073774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.073724 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:47.108014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.107979 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/701faa90-8cba-4b4e-a189-ab1db587a0c0-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:47.108248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.108225 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rlk\" (UniqueName: \"kubernetes.io/projected/701faa90-8cba-4b4e-a189-ab1db587a0c0-kube-api-access-g5rlk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:47.484083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.482807 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.534736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.534695 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:53:47.597109 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.580691 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:47.599580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.599532 5733 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.743133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.735176 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:47.743133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.735249 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:47.743133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.735652 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:53:47.915082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.889133 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:47.915082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.889193 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:47.919076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.915770 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:47.919076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:47.915834 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:48.379119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.379078 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:48.657712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.657577 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:48.689674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.689176 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:48.689674 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.689254 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:48.862429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.861510 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:53:48.862429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.861573 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 19:53:48.863068 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.862887 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:48.907241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.907196 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:53:48.907386 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:48.907297 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 19:53:49.168082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.167150 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:49.209078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.208794 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:49.691760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.691415 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 19:53:49.691760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.691462 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 19:53:49.835078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.834189 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:49.864496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.864458 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:49.865431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.865407 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:49.866193 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:49.866168 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 19:53:50.009148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.007201 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:50.009148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.009134 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:50.014082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.010629 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 19:53:50.124081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.123997 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:53:50.693118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.693092 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:50.693703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.693679 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:50.694483 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:50.694465 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 19:53:51.054847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:51.054761 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:51.296243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:51.296215 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:52.415830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:52.415792 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:53.341811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:53.341719 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.335200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:54.335158 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.137150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.137105 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 19:53:55.138163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.138143 5733 cpu_manager.go:414] "containerMap: removing container" podUID="701faa90-8cba-4b4e-a189-ab1db587a0c0" containerName="revoke-keys" Oct 30 19:53:55.138266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.138165 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701faa90-8cba-4b4e-a189-ab1db587a0c0" containerName="revoke-keys" Oct 30 19:53:55.138328 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.138315 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="701faa90-8cba-4b4e-a189-ab1db587a0c0" containerName="revoke-keys" Oct 30 19:53:55.143272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.143157 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.173897 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.173866 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 19:53:55.220098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220042 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220251 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220129 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220459 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220432 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220486 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220585 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7bdx\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220660 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220625 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220711 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220655 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220762 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.220834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.220808 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322633 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322597 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7bdx\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322647 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322677 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322705 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322747 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.322815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322787 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.322850 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323100 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323159 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323203 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323324 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323308 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323408 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323587 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.323839 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.323760 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.325905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.325876 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.326463 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.326438 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.339120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.339093 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 19:53:55.350120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.350094 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.360791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.360760 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7bdx\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.373123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.369386 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.376811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.376774 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 19:53:55.424493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424413 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.424493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424455 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.424670 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424530 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.424775 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424694 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxf8v\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.424875 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424853 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.425002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.424982 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.425045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.425020 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.425152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.425134 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.425279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.425261 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.477917 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.477875 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:55.526848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.526810 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.526848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.526851 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.526900 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.526986 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527036 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527079 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527117 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527246 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxf8v\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527287 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527371 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.527975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.527945 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.528502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.528246 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.528502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.528340 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.528502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.528457 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.530683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.530660 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.532392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.532371 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.544090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.544029 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxf8v\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.545002 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.544979 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\") pod \"e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.602353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.601703 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.693402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.693314 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:55.780170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.780128 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 19:53:55.782546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.782515 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:55.782653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.782587 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:55.782653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.782633 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:55.896490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.896453 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z"] Oct 30 19:53:55.901636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.901611 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:55.917462 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.917429 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z"] Oct 30 19:53:55.919264 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.918506 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:55.919264 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:55.919256 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:53:56.004449 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:53:56.004414 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655268c1_99fe_42d1_9545_fa137a8dd02c.slice/crio-e5c152858637f22665964706d0fcb8a4e6fea6337849b40e6bc8db2d2a101e2f WatchSource:0}: Error finding container e5c152858637f22665964706d0fcb8a4e6fea6337849b40e6bc8db2d2a101e2f: Status 404 returned error can't find the container with id e5c152858637f22665964706d0fcb8a4e6fea6337849b40e6bc8db2d2a101e2f Oct 30 19:53:56.006451 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.006425 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 19:53:56.006898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.006867 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.006987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.006946 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.007047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.006992 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.053458 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.053425 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmtt\" (UniqueName: \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.053759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.053733 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.053960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.053940 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.076876 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.076704 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 19:53:56.083747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.083028 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.092437 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.092400 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 19:53:56.162679 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.162407 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmtt\" (UniqueName: \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.169638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.169607 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.169808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.169656 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.169808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.169766 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.169960 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.169942 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.170087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.170069 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.170165 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.170139 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.170220 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.170173 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8zz\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.171796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.171769 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.172079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.172034 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.172229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.172214 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.172486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.172470 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.176664 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.176640 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.178198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.178178 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.190757 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.190724 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmtt\" (UniqueName: \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\") pod \"maas-api-revoke-sub-9b02b572-4sc6z\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.200513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.200473 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" event={"ID":"655268c1-99fe-42d1-9545-fa137a8dd02c","Type":"ContainerStarted","Data":"12ae1382066279728d6d959ea5d012ea147638af07d101683a46d2e50b37ae93"} Oct 30 19:53:56.200761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.200739 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" event={"ID":"655268c1-99fe-42d1-9545-fa137a8dd02c","Type":"ContainerStarted","Data":"e5c152858637f22665964706d0fcb8a4e6fea6337849b40e6bc8db2d2a101e2f"} Oct 30 19:53:56.202966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.202942 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" event={"ID":"4aae276e-c30b-44e3-af8d-d88aa479cfb4","Type":"ContainerStarted","Data":"e98008f879ac4beff479e8c78adfd03b818726f79887cd84a36feb340a6d2684"} Oct 30 19:53:56.203157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.203140 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" event={"ID":"4aae276e-c30b-44e3-af8d-d88aa479cfb4","Type":"ContainerStarted","Data":"c72da464285303ada699e44d9c5e7ffdb6e41af5ceb6693a8f6e12d75e36027e"} Oct 30 19:53:56.257120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.257007 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:53:56.266654 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.266594 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podStartSLOduration=1.266573087 podStartE2EDuration="1.266573087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.260933857 +0000 UTC m=+1396.328769425" watchObservedRunningTime="2026-10-30 19:53:56.266573087 +0000 UTC m=+1396.334408655" Oct 30 19:53:56.274678 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.274643 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.275073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.275014 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.275243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.275226 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8zz\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.275731 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.275712 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.277748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.277623 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.278192 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.278136 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.278192 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.278143 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.278377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.278360 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.278503 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.278488 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.278920 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.278808 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.279136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.279121 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.279585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.279467 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.279700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.279537 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.279811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.279798 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.280229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.279899 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.281317 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.281229 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.306091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.302486 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.306091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.305573 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8zz\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\") pod \"e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.408863 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.408604 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:56.478625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.478595 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:53:56.479894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.479867 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:53:56.480007 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.479911 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:53:56.573907 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.571491 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podStartSLOduration=1.571469276 podStartE2EDuration="1.571469276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.320677373 +0000 UTC m=+1396.388512943" watchObservedRunningTime="2026-10-30 19:53:56.571469276 +0000 UTC m=+1396.639304848" Oct 30 19:53:56.577189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.576971 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z"] Oct 30 19:53:56.586806 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.586740 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:56.599369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.599342 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:53:56.693973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.693886 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:53:56.765680 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.765646 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 19:53:56.848810 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.848769 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.848957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.848855 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.848957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.848887 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:53:56.932922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:56.932826 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:57.275081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.266105 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" event={"ID":"1a49222b-1864-4b1f-bf27-15e207bc17f5","Type":"ContainerStarted","Data":"a1dc7fe9a61003310f76426a30fb6ae8c9d5af188a4d9d6eb9031e015f8badbc"} Oct 30 19:53:57.275081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.266172 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" event={"ID":"1a49222b-1864-4b1f-bf27-15e207bc17f5","Type":"ContainerStarted","Data":"0ea7ba90d58c332853301177e28380e22f83deabfbf221809aeb6282139a963e"} Oct 30 19:53:57.278436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.275684 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" event={"ID":"58574de6-b277-4383-bc54-a9292a4191a9","Type":"ContainerStarted","Data":"fe1945927c90b8c4e97387e7edaa317f6d2ce3e1f992d01a8f4f8d3cea35929c"} Oct 30 19:53:57.278436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.275745 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" event={"ID":"58574de6-b277-4383-bc54-a9292a4191a9","Type":"ContainerStarted","Data":"d47cd215bc0a81cee3820d13d469cb90599092f3868b0577780f42f5a9083b5b"} Oct 30 19:53:57.303234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.303153 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podStartSLOduration=1.303139357 podStartE2EDuration="1.303139357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:57.298412883 +0000 UTC m=+1397.366248454" watchObservedRunningTime="2026-10-30 19:53:57.303139357 +0000 UTC m=+1397.370974924" Oct 30 19:53:57.408837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.408807 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:53:57.437137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.434185 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:53:57.437137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.434265 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:53:57.480082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.479252 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:53:57.480082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.479321 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:53:57.696234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.696142 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:53:57.696234 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:57.696215 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:53:58.316092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.315196 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:58.411085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.410588 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:53:58.411085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.410652 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:53:58.504671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.493143 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:53:58.504671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.493209 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:53:58.533087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.529788 5733 generic.go:356] "Generic (PLEG): container finished" podID="58574de6-b277-4383-bc54-a9292a4191a9" containerID="fe1945927c90b8c4e97387e7edaa317f6d2ce3e1f992d01a8f4f8d3cea35929c" exitCode=0 Oct 30 19:53:58.538094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.532109 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" event={"ID":"58574de6-b277-4383-bc54-a9292a4191a9","Type":"ContainerDied","Data":"fe1945927c90b8c4e97387e7edaa317f6d2ce3e1f992d01a8f4f8d3cea35929c"} Oct 30 19:53:58.733082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.729163 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:53:58.733082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:58.729229 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:53:59.409697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.409664 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:53:59.409860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.409721 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:53:59.486083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.482463 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:53:59.486083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.482535 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:53:59.490849 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:53:59.490783 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:53:59.694954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.694441 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:53:59.694954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:53:59.694505 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:54:00.455274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.455220 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:54:00.455579 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.455534 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:54:00.502914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.499644 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:00.502914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.499699 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:54:00.527229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.527146 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:00.609323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.609273 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:00.626920 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.626885 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:54:00.764506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.740010 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:00.764506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.740091 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:54:00.764506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.750715 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:00.788597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.774660 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\") pod \"58574de6-b277-4383-bc54-a9292a4191a9\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " Oct 30 19:54:00.788597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.774875 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\" (UniqueName: \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\") pod \"58574de6-b277-4383-bc54-a9292a4191a9\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " Oct 30 19:54:00.788597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.775039 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\") pod \"58574de6-b277-4383-bc54-a9292a4191a9\" (UID: \"58574de6-b277-4383-bc54-a9292a4191a9\") " Oct 30 19:54:00.788597 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.776277 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca" pod "58574de6-b277-4383-bc54-a9292a4191a9" (UID: "58574de6-b277-4383-bc54-a9292a4191a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:00.892417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.892379 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.974080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.973957 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt" pod "58574de6-b277-4383-bc54-a9292a4191a9" (UID: "58574de6-b277-4383-bc54-a9292a4191a9"). InnerVolumeSpecName "kube-api-access-tpmtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:01.002095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:00.996589 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca" pod "58574de6-b277-4383-bc54-a9292a4191a9" (UID: "58574de6-b277-4383-bc54-a9292a4191a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:01.050084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.024019 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58574de6-b277-4383-bc54-a9292a4191a9-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.139156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.139118 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpmtt\" (UniqueName: \"kubernetes.io/projected/58574de6-b277-4383-bc54-a9292a4191a9-kube-api-access-tpmtt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:01.139448 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.074486 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" event={"ID":"58574de6-b277-4383-bc54-a9292a4191a9","Type":"ContainerDied","Data":"d47cd215bc0a81cee3820d13d469cb90599092f3868b0577780f42f5a9083b5b"} Oct 30 19:54:01.139650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.139631 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d47cd215bc0a81cee3820d13d469cb90599092f3868b0577780f42f5a9083b5b" Oct 30 19:54:01.139763 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.122299 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.140167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.064016 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.144079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.074722 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z" Oct 30 19:54:01.171530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.171490 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.242083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.209841 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.362951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.361861 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.415350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.414909 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:54:01.415350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.414969 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:54:01.499713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.499245 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.506078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.500199 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:01.506078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.500241 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:54:01.609085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.581423 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:01.708627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.708539 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:01.708886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.708856 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:54:01.753470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.751085 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:01.766363 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.766332 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:01.931089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:01.930178 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:02.212552 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.212508 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:02.393088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.380296 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:02.393088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.385196 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:02.471139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.445786 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:54:02.506728 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.506678 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:54:02.506997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.506684 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:54:02.533082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.521678 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:54:02.533082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.523960 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:02.533082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.524006 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:54:02.571146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.559692 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:02.591585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.574881 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:54:02.776871 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.776770 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:02.777491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.777441 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:54:02.782232 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.782209 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:02.783605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.783584 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:02.784885 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.784020 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:02.785081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.785048 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:02.794375 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.794341 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:02.838836 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.838804 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:02.879283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.879248 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:02.895635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:02.895605 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:03.257091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.249687 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:03.409841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.409804 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:54:03.412201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.410155 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:54:03.478596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.478562 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 19:54:03.479307 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.478832 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 19:54:03.694083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.694019 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:54:03.694292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:03.694090 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Oct 30 19:54:04.410703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.410163 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:54:04.410703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.410231 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 19:54:04.576019 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.575832 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:54:04.577350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.577327 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:54:04.584521 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.583458 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 19:54:04.700685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.700600 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:54:04.701490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.701460 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:54:04.703276 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.703242 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 19:54:04.843081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:04.841421 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:05.655736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:05.655705 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:54:05.657345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:05.657319 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:54:05.658309 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:05.658291 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 19:54:06.526089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:06.524145 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:08.073953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.072206 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:08.332817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.332716 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.476081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.471191 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.496094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.492467 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.527347 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.527301 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.612169 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.594365 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.736784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.736729 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.824082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.811039 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.834200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.834154 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:08.870432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.870333 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.888210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.888158 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:08.908235 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:08.908188 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.073235 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.073176 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.074198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.074172 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:09.395078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.390930 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.395078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.391754 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:09.480125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.472168 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.480125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.472914 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:09.507146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.493276 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:54:09.507146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.493715 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:09.558116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.554503 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:54:09.721573 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.620694 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.723126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.646702 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.746160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.722277 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:09.746454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.746434 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:09.759086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.743458 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.759907 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.759886 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:09.815439 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.815390 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:09.827335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.827297 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:09.858296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.858252 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:09.859806 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.859780 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:09.923474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.923427 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.032876 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.987289 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:10.032876 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.924125 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.032876 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:09.995503 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:10.199087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.179993 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.235114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.201752 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.406774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.406727 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.489256 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.489212 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.522824 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.522781 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:54:10.730153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.727240 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:10.730153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.729956 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:10.767082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.766545 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:54:10.835083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.834148 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:10.880081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.865123 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:10.991525 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:10.991425 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.008094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.002354 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.195226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.181248 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.209241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.209193 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.209953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.209922 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.457084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.445754 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:11.511484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.472174 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.533210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.533160 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.533547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.533520 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.704080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.633528 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:11.822366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.822262 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.863968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.863929 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:11.882117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.875302 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:11.883345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.881375 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:11.912644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:11.911122 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:12.036908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.035163 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:12.063510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.063468 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:12.233145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.221714 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:12.260932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.260891 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.487979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.487506 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:12.494962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.494939 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:12.495605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.495587 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:12.496784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.496767 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:12.496968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.496956 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:12.499628 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.499243 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:12.510999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.505202 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:12.538012 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.537986 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:12.541089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.540893 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:54:12.569993 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.569945 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:12.576676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.576626 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:12.617086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:12.609449 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:13.615817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.615776 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.702953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.702919 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx"] Oct 30 19:54:13.703677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.703656 5733 cpu_manager.go:414] "containerMap: removing container" podUID="58574de6-b277-4383-bc54-a9292a4191a9" containerName="revoke-keys" Oct 30 19:54:13.703677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.703677 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58574de6-b277-4383-bc54-a9292a4191a9" containerName="revoke-keys" Oct 30 19:54:13.703894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.703880 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="58574de6-b277-4383-bc54-a9292a4191a9" containerName="revoke-keys" Oct 30 19:54:13.709471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.709445 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.718100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.718040 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx"] Oct 30 19:54:13.718345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.718322 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:13.718688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.718661 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:54:13.822965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.822931 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.823262 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.823233 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkzn\" (UniqueName: \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.823384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.823284 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.924640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.924548 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkzn\" (UniqueName: \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.924640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.924592 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.924868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.924728 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.925638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.925611 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.928104 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.928080 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:13.938890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:13.938862 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkzn\" (UniqueName: \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\") pod \"maas-api-revoke-sub-6bfd631a-gh8rx\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:14.084316 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.084280 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:14.409834 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:14.409792 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6947777b_f683_42d2_a03f_3b0c838bef21.slice/crio-9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879 WatchSource:0}: Error finding container 9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879: Status 404 returned error can't find the container with id 9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879 Oct 30 19:54:14.410486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.410456 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx"] Oct 30 19:54:14.532076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.526125 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.650353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.650301 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd"] Oct 30 19:54:14.657343 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.657266 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.666310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.666285 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd"] Oct 30 19:54:14.749025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.748993 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmcq\" (UniqueName: \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.749221 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.749145 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.749349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.749329 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.854789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.851253 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.854789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.851505 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmcq\" (UniqueName: \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.854789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.851585 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.854789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.852946 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.854789 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.854790 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.865635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.865611 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmcq\" (UniqueName: \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\") pod \"maas-api-revoke-sub-de07ab40-h5hqd\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:14.975970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:14.975931 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:15.226223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:15.226120 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" event={"ID":"6947777b-f683-42d2-a03f-3b0c838bef21","Type":"ContainerStarted","Data":"1e89fea7b5388508d0c4711700312ba047f8b15c153b497725162a8f4ee64694"} Oct 30 19:54:15.226426 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:15.226237 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" event={"ID":"6947777b-f683-42d2-a03f-3b0c838bef21","Type":"ContainerStarted","Data":"9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879"} Oct 30 19:54:15.250677 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:15.250622 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" podStartSLOduration=2.250607987 podStartE2EDuration="2.250607987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:15.246611665 +0000 UTC m=+1415.314447228" watchObservedRunningTime="2026-10-30 19:54:15.250607987 +0000 UTC m=+1415.318443554" Oct 30 19:54:15.274946 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:15.274914 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57288644_cfa7_44a9_bf39_43ecd96d7939.slice/crio-ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385 WatchSource:0}: Error finding container ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385: Status 404 returned error can't find the container with id ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385 Oct 30 19:54:15.275490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:15.275464 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd"] Oct 30 19:54:15.506134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:15.506008 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.238219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.238183 5733 generic.go:356] "Generic (PLEG): container finished" podID="6947777b-f683-42d2-a03f-3b0c838bef21" containerID="1e89fea7b5388508d0c4711700312ba047f8b15c153b497725162a8f4ee64694" exitCode=0 Oct 30 19:54:16.238598 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.238324 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" event={"ID":"6947777b-f683-42d2-a03f-3b0c838bef21","Type":"ContainerDied","Data":"1e89fea7b5388508d0c4711700312ba047f8b15c153b497725162a8f4ee64694"} Oct 30 19:54:16.240155 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.240132 5733 generic.go:356] "Generic (PLEG): container finished" podID="57288644-cfa7-44a9-bf39-43ecd96d7939" containerID="1e009ef97ae8f3ad95e4903f5777a1e58cc5d9d8c27ff6115b3404517f0891de" exitCode=0 Oct 30 19:54:16.240311 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.240282 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" event={"ID":"57288644-cfa7-44a9-bf39-43ecd96d7939","Type":"ContainerDied","Data":"1e009ef97ae8f3ad95e4903f5777a1e58cc5d9d8c27ff6115b3404517f0891de"} Oct 30 19:54:16.240359 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.240343 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" event={"ID":"57288644-cfa7-44a9-bf39-43ecd96d7939","Type":"ContainerStarted","Data":"ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385"} Oct 30 19:54:16.489744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:16.489651 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.493852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.493823 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:17.559506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.559383 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:17.596927 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.596891 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.615368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.615337 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\") pod \"6947777b-f683-42d2-a03f-3b0c838bef21\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " Oct 30 19:54:17.615555 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.615537 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\" (UniqueName: \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\") pod \"6947777b-f683-42d2-a03f-3b0c838bef21\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " Oct 30 19:54:17.615691 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.615676 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\") pod \"6947777b-f683-42d2-a03f-3b0c838bef21\" (UID: \"6947777b-f683-42d2-a03f-3b0c838bef21\") " Oct 30 19:54:17.616598 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.616572 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca" pod "6947777b-f683-42d2-a03f-3b0c838bef21" (UID: "6947777b-f683-42d2-a03f-3b0c838bef21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:17.620372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.620160 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn" pod "6947777b-f683-42d2-a03f-3b0c838bef21" (UID: "6947777b-f683-42d2-a03f-3b0c838bef21"). InnerVolumeSpecName "kube-api-access-2qkzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:17.621461 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.621436 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca" pod "6947777b-f683-42d2-a03f-3b0c838bef21" (UID: "6947777b-f683-42d2-a03f-3b0c838bef21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:17.717737 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.717700 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\") pod \"57288644-cfa7-44a9-bf39-43ecd96d7939\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " Oct 30 19:54:17.717968 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.717951 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\" (UniqueName: \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\") pod \"57288644-cfa7-44a9-bf39-43ecd96d7939\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " Oct 30 19:54:17.718044 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718000 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\") pod \"57288644-cfa7-44a9-bf39-43ecd96d7939\" (UID: \"57288644-cfa7-44a9-bf39-43ecd96d7939\") " Oct 30 19:54:17.718397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718369 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca" pod "57288644-cfa7-44a9-bf39-43ecd96d7939" (UID: "57288644-cfa7-44a9-bf39-43ecd96d7939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:17.718754 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718732 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:17.718852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718757 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:17.718852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718772 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6947777b-f683-42d2-a03f-3b0c838bef21-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:17.718852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.718786 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkzn\" (UniqueName: \"kubernetes.io/projected/6947777b-f683-42d2-a03f-3b0c838bef21-kube-api-access-2qkzn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:17.720091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.720052 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq" pod "57288644-cfa7-44a9-bf39-43ecd96d7939" (UID: "57288644-cfa7-44a9-bf39-43ecd96d7939"). InnerVolumeSpecName "kube-api-access-jdmcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:17.720312 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.720285 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca" pod "57288644-cfa7-44a9-bf39-43ecd96d7939" (UID: "57288644-cfa7-44a9-bf39-43ecd96d7939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:17.820021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.819927 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmcq\" (UniqueName: \"kubernetes.io/projected/57288644-cfa7-44a9-bf39-43ecd96d7939-kube-api-access-jdmcq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:17.820021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:17.819971 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57288644-cfa7-44a9-bf39-43ecd96d7939-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:18.260543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.260512 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" Oct 30 19:54:18.260543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.260531 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx" event={"ID":"6947777b-f683-42d2-a03f-3b0c838bef21","Type":"ContainerDied","Data":"9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879"} Oct 30 19:54:18.260758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.260591 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b1b777c4804ca4bf27ca5b4acf49cead640b1bc820b06bb7e03e7a3cd39e879" Oct 30 19:54:18.262163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.262142 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" Oct 30 19:54:18.262296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.262189 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd" event={"ID":"57288644-cfa7-44a9-bf39-43ecd96d7939","Type":"ContainerDied","Data":"ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385"} Oct 30 19:54:18.262296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.262232 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba4f82ef9f3b173d019f08460af327bc169d0152c9dd2ee8264239618b629385" Oct 30 19:54:18.818105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:18.818032 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:19.753957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:19.753900 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:20.716279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:20.716240 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:21.506177 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:21.506135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.525723 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:22.525361 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.482943 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:23.482906 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.418675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:24.418638 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:25.773650 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.773615 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 19:54:25.774688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774663 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6947777b-f683-42d2-a03f-3b0c838bef21" containerName="revoke-keys" Oct 30 19:54:25.774769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774692 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6947777b-f683-42d2-a03f-3b0c838bef21" containerName="revoke-keys" Oct 30 19:54:25.774769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774745 5733 cpu_manager.go:414] "containerMap: removing container" podUID="57288644-cfa7-44a9-bf39-43ecd96d7939" containerName="revoke-keys" Oct 30 19:54:25.774769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774754 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57288644-cfa7-44a9-bf39-43ecd96d7939" containerName="revoke-keys" Oct 30 19:54:25.774934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774921 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="57288644-cfa7-44a9-bf39-43ecd96d7939" containerName="revoke-keys" Oct 30 19:54:25.774966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.774949 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6947777b-f683-42d2-a03f-3b0c838bef21" containerName="revoke-keys" Oct 30 19:54:25.780810 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.780776 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.828084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.820131 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 19:54:25.858266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.858228 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.858438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.858350 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.858976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.858947 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859127 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859170 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859224 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859209 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859334 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859234 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859397 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.859516 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.859441 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs2wj\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.861351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.861318 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:25.960963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.960927 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961080 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961110 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961139 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961163 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961263 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961294 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs2wj\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961345 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961391 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961458 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961481 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961563 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.961889 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.961868 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.962282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.962255 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.963932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.963643 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.964711 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.964676 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.975698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.975672 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:25.975915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:25.975889 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs2wj\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:26.147805 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.147767 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:26.466688 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:26.466651 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ecb511_0de4_455d_b9f8_9faea4093ff8.slice/crio-595e169146e7a9c5c5fb30655b79e9b402355af396ef5491fa3891fd64f96d91 WatchSource:0}: Error finding container 595e169146e7a9c5c5fb30655b79e9b402355af396ef5491fa3891fd64f96d91: Status 404 returned error can't find the container with id 595e169146e7a9c5c5fb30655b79e9b402355af396ef5491fa3891fd64f96d91 Oct 30 19:54:26.469385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.469341 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:26.469493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.469427 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:26.469493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.469467 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:26.471111 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.471043 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 19:54:26.617711 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.617671 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 19:54:26.623673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.623636 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.672528 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673350 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673466 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673557 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673623 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673684 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxln\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673724 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673795 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673826 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.674135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.673864 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.775884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.775788 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.775884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.775848 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.775892 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.775981 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.776215 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.776311 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.776365 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.776415 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxln\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.776645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.776461 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.777569 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.777536 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.777671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.777599 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.778440 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.778418 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.778605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.778428 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.778914 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.778891 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.780886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.780865 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.781278 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.781257 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.804181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.804154 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxln\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.805018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.804995 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\") pod \"e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:26.947763 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:26.947722 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:27.086984 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.086956 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn"] Oct 30 19:54:27.092655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.092635 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.093758 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.093720 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn"] Oct 30 19:54:27.101304 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.101281 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:54:27.102015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.101486 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:27.183283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.183245 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm66\" (UniqueName: \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.183483 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.183303 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.183641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.183613 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.285175 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.285080 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.285361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.285243 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm66\" (UniqueName: \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.285361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.285284 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.286167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.286141 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.287759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.287740 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.307579 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:27.307552 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c8d7fd_e7fd_462e_99ab_ef682af23939.slice/crio-0cab5f1c4b560b6c2219271b55c05baf6d89af710e737f1a9cb946bd4b5825e7 WatchSource:0}: Error finding container 0cab5f1c4b560b6c2219271b55c05baf6d89af710e737f1a9cb946bd4b5825e7: Status 404 returned error can't find the container with id 0cab5f1c4b560b6c2219271b55c05baf6d89af710e737f1a9cb946bd4b5825e7 Oct 30 19:54:27.318096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.310390 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:27.318096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.310485 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:27.318096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.310495 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 19:54:27.318096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.310527 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:27.335271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.330634 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm66\" (UniqueName: \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\") pod \"maas-api-revoke-sub-92e2d96d-w7gpn\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.360757 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.360721 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" event={"ID":"20c8d7fd-e7fd-462e-99ab-ef682af23939","Type":"ContainerStarted","Data":"0cab5f1c4b560b6c2219271b55c05baf6d89af710e737f1a9cb946bd4b5825e7"} Oct 30 19:54:27.362823 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.362793 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" event={"ID":"70ecb511-0de4-455d-b9f8-9faea4093ff8","Type":"ContainerStarted","Data":"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706"} Oct 30 19:54:27.362947 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.362847 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" event={"ID":"70ecb511-0de4-455d-b9f8-9faea4093ff8","Type":"ContainerStarted","Data":"595e169146e7a9c5c5fb30655b79e9b402355af396ef5491fa3891fd64f96d91"} Oct 30 19:54:27.398987 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.398936 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podStartSLOduration=2.39892111 podStartE2EDuration="2.39892111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:27.394800096 +0000 UTC m=+1427.462635661" watchObservedRunningTime="2026-10-30 19:54:27.39892111 +0000 UTC m=+1427.466756677" Oct 30 19:54:27.410192 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.410161 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:27.832485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.832385 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn"] Oct 30 19:54:27.852800 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:27.852768 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34acfa86_26f8_4511_8dc8_5060a185b612.slice/crio-2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4 WatchSource:0}: Error finding container 2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4: Status 404 returned error can't find the container with id 2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4 Oct 30 19:54:27.925644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:27.925599 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.148396 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.148358 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:28.150025 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.149994 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:28.150197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.150087 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:28.373004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.372962 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" event={"ID":"34acfa86-26f8-4511-8dc8-5060a185b612","Type":"ContainerStarted","Data":"308687b11b045687e7b67987504d93e9627da35f9e918779816fc4ef541a2ea7"} Oct 30 19:54:28.373215 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.373025 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" event={"ID":"34acfa86-26f8-4511-8dc8-5060a185b612","Type":"ContainerStarted","Data":"2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4"} Oct 30 19:54:28.374768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.374737 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" event={"ID":"20c8d7fd-e7fd-462e-99ab-ef682af23939","Type":"ContainerStarted","Data":"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756"} Oct 30 19:54:28.404164 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.404069 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" podStartSLOduration=1.40404422 podStartE2EDuration="1.40404422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:28.40058793 +0000 UTC m=+1428.468423499" watchObservedRunningTime="2026-10-30 19:54:28.40404422 +0000 UTC m=+1428.471879788" Oct 30 19:54:28.951086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.949474 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:28.965082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.963188 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:28.965082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:28.963256 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:29.069077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.061602 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.157083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.155617 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:29.157083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.155694 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:29.385003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.384971 5733 generic.go:356] "Generic (PLEG): container finished" podID="34acfa86-26f8-4511-8dc8-5060a185b612" containerID="308687b11b045687e7b67987504d93e9627da35f9e918779816fc4ef541a2ea7" exitCode=0 Oct 30 19:54:29.385189 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.385090 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" event={"ID":"34acfa86-26f8-4511-8dc8-5060a185b612","Type":"ContainerDied","Data":"308687b11b045687e7b67987504d93e9627da35f9e918779816fc4ef541a2ea7"} Oct 30 19:54:29.402795 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:54:29.402769 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:54:29.437085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.434727 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podStartSLOduration=3.434706925 podStartE2EDuration="3.434706925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:28.452319242 +0000 UTC m=+1428.520154807" watchObservedRunningTime="2026-10-30 19:54:29.434706925 +0000 UTC m=+1429.502542498" Oct 30 19:54:29.951299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.951260 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:29.962196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:29.962129 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:30.152485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.152446 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:30.152740 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.152708 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:30.595286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.582918 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:30.975561 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.975514 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:30.976036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.975575 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:30.986255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:30.986229 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:31.103302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.103268 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\" (UniqueName: \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\") pod \"34acfa86-26f8-4511-8dc8-5060a185b612\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " Oct 30 19:54:31.103499 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.103485 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\") pod \"34acfa86-26f8-4511-8dc8-5060a185b612\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " Oct 30 19:54:31.103568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.103556 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\") pod \"34acfa86-26f8-4511-8dc8-5060a185b612\" (UID: \"34acfa86-26f8-4511-8dc8-5060a185b612\") " Oct 30 19:54:31.104400 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.104365 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca" pod "34acfa86-26f8-4511-8dc8-5060a185b612" (UID: "34acfa86-26f8-4511-8dc8-5060a185b612"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:31.106503 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.106474 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66" pod "34acfa86-26f8-4511-8dc8-5060a185b612" (UID: "34acfa86-26f8-4511-8dc8-5060a185b612"). InnerVolumeSpecName "kube-api-access-bdm66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:31.108513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.108451 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca" pod "34acfa86-26f8-4511-8dc8-5060a185b612" (UID: "34acfa86-26f8-4511-8dc8-5060a185b612"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:31.157079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.156288 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.157079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.156362 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:31.205774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.205720 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:31.205774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.205770 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdm66\" (UniqueName: \"kubernetes.io/projected/34acfa86-26f8-4511-8dc8-5060a185b612-kube-api-access-bdm66\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:31.206015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.205787 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34acfa86-26f8-4511-8dc8-5060a185b612-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:31.404271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.404194 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" event={"ID":"34acfa86-26f8-4511-8dc8-5060a185b612","Type":"ContainerDied","Data":"2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4"} Oct 30 19:54:31.404271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.404245 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2968c71b5f04bd0b7a06550b639adf4b51bd40285d6a12f79aafc503994dcbe4" Oct 30 19:54:31.404452 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.404295 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn" Oct 30 19:54:31.957082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.952929 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:31.957082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.953001 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:31.989112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:31.985657 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:32.167085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:32.166390 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:32.167085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:32.166470 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:32.950505 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:32.950466 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:32.950823 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:32.950793 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:33.149471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:33.149426 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.149951 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:33.149524 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 19:54:33.374051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:33.373658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.953613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:33.953572 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:33.953915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:33.953883 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:34.406617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.406574 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:34.408721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.408685 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:34.410340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.410310 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 19:54:34.467705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.467674 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:34.949591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.949553 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 19:54:34.949814 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:34.949624 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 19:54:35.733688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:35.733601 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:35.954717 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:35.954685 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:35.955154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:35.955133 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:35.956008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:35.955987 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 19:54:37.007480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:37.007439 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.875257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:38.875192 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:38.919526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:38.919461 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.011402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.010186 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.015909 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.015160 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.150836 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.150738 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.197092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.194727 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:39.321075 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.320925 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.438427 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.437175 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.453587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.452416 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.543667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.541170 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.543667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.542407 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.650176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.648197 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.740148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.740000 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:39.772513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:39.772466 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.122457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.121935 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:40.260112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.148801 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:54:40.260112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.243176 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.361606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.157490 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:40.432142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.419808 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:40.664084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.639432 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.959094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.749423 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.959729 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.753363 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.959907 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.836215 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:54:40.959994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.836379 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:40.960089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:40.836555 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:54:41.062083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.029384 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:41.110086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.079169 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 19:54:41.259613 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.259498 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 19:54:41.558138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.538732 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:41.558138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.540149 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:54:41.559964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.559923 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:41.752977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:41.748706 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.004497 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:54:42.003533 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.829s" Oct 30 19:54:42.023085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.005449 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069404 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069470 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069511 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069546 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069582 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069619 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069655 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069689 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:42.071113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.069725 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:42.072080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.055593 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.096329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.096231 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.256364 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.055650 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.320255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.320210 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:42.402113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.383169 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.402113 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.401912 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:42.402700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.402662 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:42.441164 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.441121 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 19:54:42.453105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.442464 5733 cpu_manager.go:414] "containerMap: removing container" podUID="34acfa86-26f8-4511-8dc8-5060a185b612" containerName="revoke-keys" Oct 30 19:54:42.453419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.453389 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34acfa86-26f8-4511-8dc8-5060a185b612" containerName="revoke-keys" Oct 30 19:54:42.458077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.453813 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="34acfa86-26f8-4511-8dc8-5060a185b612" containerName="revoke-keys" Oct 30 19:54:42.512519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.512420 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.518202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.518159 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Oct 30 19:54:42.536083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.527828 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:54:42.836091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.745261 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:54:42.836091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.761914 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:42.836091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.780913 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:54:42.836091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.781923 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:42.897219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.897179 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.004080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:42.975185 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.072497 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.072451 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.287084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.258158 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.386488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.386429 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.447149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.395151 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.447149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.410176 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:43.447149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.410214 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:43.528145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.520321 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661330 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661433 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661482 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661514 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661551 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25hc\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661693 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661806 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661832 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.664132 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.661886 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.726321 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.713021 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:43.755154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.733574 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 19:54:43.788478 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.788383 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 19:54:43.823964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.823930 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 19:54:43.824239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.773141 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.824355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.773591 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.824557 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.824541 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:43.847032 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.846990 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.848622 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.848585 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:43.849832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.849805 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:54:43.850962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.850938 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.942761 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.942819 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.942872 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.942957 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.943079 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.943135 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.943172 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:43.945081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:43.943231 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25hc\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.040973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.011355 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.069513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.059946 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.069513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.060539 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.107084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.099369 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.106436 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.171830 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25hc\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.183401 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.219427 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.479621 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.479696 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:44.480157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.287422 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.252187 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481635 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481669 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481699 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481837 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481871 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qxw\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.481949 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.482084 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.320979 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.287370 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\") pod \"e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.482408 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.432009 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.432085 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.432143 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:44.503003 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.432123 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:44.743313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.741214 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:44.823107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.788219 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:45.310091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.285317 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.317282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.317233 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.317595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.317577 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.317833 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.317811 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.318006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.317984 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.318235 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.318220 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.318369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.318356 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qxw\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.318563 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.318534 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.318975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.318956 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.320261 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.320238 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.320411 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.892465 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.345211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.325539 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.345535 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.892657 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:54:45.347431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.891959 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 19:54:45.398636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.999414 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.398973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.892865 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:54:45.421222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.893447 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:45.439144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:44.893154 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:54:45.504687 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.504628 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:45.618147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.529162 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:45.794520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.784858 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.858980 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.858948 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.886173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.869045 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.890020 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.889905 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 19:54:45.890595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.890567 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.286322 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.898831 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902150 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902227 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902323 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902419 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902518 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902588 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs97p\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902679 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902739 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.905118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.902778 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:45.934119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.927412 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:45.989109 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.905934 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:46.001542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.001502 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:46.002316 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:45.961339 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:46.162135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.126472 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:46.162135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.126775 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.167475 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qxw\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.175921 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.175996 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs97p\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176137 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176183 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176213 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176450 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176503 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176632 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.176727 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.177284 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.179468 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.180754 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.293572 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.293524 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:46.295494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.295466 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.306160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.306129 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.308614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.306897 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:46.352341 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.308557 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.421129 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.376162 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:46.421129 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:54:46.376246 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.028s" Oct 30 19:54:46.425096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.425047 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:46.425411 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.425395 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:46.441282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.441245 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:46.441809 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.441776 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:46.488481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.485728 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:46.497123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.491346 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:46.526556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.526524 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:46.544138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.543989 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:46.580095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.576365 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.592488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.592460 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:46.592745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.592730 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:46.595937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.595907 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.598392 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.598155 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:46.608527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.608488 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:46.613484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.613443 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:46.634127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.634103 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs97p\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\") pod \"e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.739195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.739145 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:46.862431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.859235 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 19:54:46.887204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.886726 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:46.887204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.886819 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:46.887204 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.886863 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:46.897555 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.897526 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:46.952762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.952722 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 19:54:46.969118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.968436 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:46.969118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.968524 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:46.969118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:46.968566 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:47.166745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.165581 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" event={"ID":"410ce3f8-0736-4f70-b75d-8e3ed39cbc88","Type":"ContainerStarted","Data":"717404bd8ebc3c8b9fb5010ac1f802b85c829786eac975b0479b06c647539ba3"} Oct 30 19:54:47.168706 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.168624 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" event={"ID":"81e3bbd1-0da1-4638-aaec-8bcf01b332a1","Type":"ContainerStarted","Data":"a89d0d7c6c91a6d9620dcf11adc0a064edcaa30c9c2932b14b58439d91021330"} Oct 30 19:54:47.331617 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.331355 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 19:54:47.342746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.342693 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:47.342979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.342958 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:47.343118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.343101 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:47.354141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.354113 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.577179 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:47.576964 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.191496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.191441 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" event={"ID":"81e3bbd1-0da1-4638-aaec-8bcf01b332a1","Type":"ContainerStarted","Data":"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9"} Oct 30 19:54:48.193340 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.193304 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" event={"ID":"410ce3f8-0736-4f70-b75d-8e3ed39cbc88","Type":"ContainerStarted","Data":"8f729eb7371d20010dba37714d365141485cb32a4f4957940e9c0626ff59b5ea"} Oct 30 19:54:48.195480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.195449 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" event={"ID":"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42","Type":"ContainerStarted","Data":"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728"} Oct 30 19:54:48.195646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.195499 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" event={"ID":"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42","Type":"ContainerStarted","Data":"8d2e11b9c8b2c4f27576b464220bc75c8c4007a5d035db1b6d1afcbad825d4b4"} Oct 30 19:54:48.322081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.320474 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:48.324022 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.322627 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:48.324022 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.322690 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:48.324022 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.323215 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.366095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.362866 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podStartSLOduration=7.362845368 podStartE2EDuration="7.362845368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:48.253651915 +0000 UTC m=+1448.321487495" watchObservedRunningTime="2026-10-30 19:54:48.362845368 +0000 UTC m=+1448.430680939" Oct 30 19:54:48.366095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.364529 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podStartSLOduration=7.36451294 podStartE2EDuration="7.36451294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:48.291526338 +0000 UTC m=+1448.359361909" watchObservedRunningTime="2026-10-30 19:54:48.36451294 +0000 UTC m=+1448.432348509" Oct 30 19:54:48.483513 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.474094 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podStartSLOduration=7.434032338 podStartE2EDuration="7.434032338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:48.350839029 +0000 UTC m=+1448.418674609" watchObservedRunningTime="2026-10-30 19:54:48.434032338 +0000 UTC m=+1448.501867908" Oct 30 19:54:48.585080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.575271 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.795142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.792471 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.926084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.911393 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:48.962595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.946906 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:48.962595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.946964 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:48.989506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:48.989469 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:49.063368 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.063271 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:49.063600 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.006378 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:49.269082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.213854 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:54:49.614279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.389999 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:49.614866 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.614832 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:49.615344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.422680 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:49.869403 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.869226 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:49.895214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.895165 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:49.926127 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.926087 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:49.926423 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.926392 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:49.927366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.927343 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:54:49.966200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.965093 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:49.972214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:49.972187 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.002640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.002598 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-df345a65-dockercfg-s446t\"" Oct 30 19:54:50.004858 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.004833 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-df345a65\"" Oct 30 19:54:50.069083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.005091 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:50.069083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.056164 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:50.115921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.103293 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.155001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.154817 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zqs\" (UniqueName: \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.186237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.186194 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 19:54:50.186959 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.186933 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.372896 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.372849 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:54:50.381690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.381655 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:54:50.475134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.386011 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:50.475134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.473856 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.475134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.418112 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:50.475134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.474143 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:50.475134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.445185 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.438107 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.456160 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.445501 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.482235 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62s7\" (UniqueName: \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.482298 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.482372 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zqs\" (UniqueName: \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:50.486087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.410032 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:50.517567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.494149 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.517567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.501845 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.717447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.717400 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:50.850005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.843747 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.874415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.874363 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.894417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.894217 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:54:50.894954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.894929 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:50.900337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.900287 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:50.972074 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.972016 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:50.985158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:50.974174 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.167138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.165236 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:51.308213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.255601 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:51.308213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.257564 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:51.308213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.257612 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:51.308213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.289137 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62s7\" (UniqueName: \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:51.308213 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.289222 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:51.350112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.343319 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:51.375142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.375106 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:51.376080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.376033 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:51.376289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.347111 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:51.384257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.384221 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:51.384549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.384524 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:51.513138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.493349 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.513138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.494198 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:51.536394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.515939 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.564363 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.564329 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:51.598081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.566362 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.602917 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.609500 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.609531 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.610536 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.610907 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:51.612092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.610988 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:51.618370 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.618343 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zqs\" (UniqueName: \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\") pod \"payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:51.833136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.815675 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:52.022081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.886047 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.052652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.050239 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.052652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.924613 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62s7\" (UniqueName: \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\") pod \"payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:52.052652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.937428 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:52.052652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.050411 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:52.052652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.980281 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.114294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:51.896346 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.114294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.078566 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:52.114294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.078677 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:52.177888 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.177856 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:52.178181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.178145 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:52.189110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.188382 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:52.251207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.251154 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.268367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.268306 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.378337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.378243 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:52.542382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.378560 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.542770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.438667 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.544130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.529291 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.545414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.542251 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:52.566162 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.529326 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:54:52.751444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.610618 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:52.759980 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.610663 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:54:52.771881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.664402 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.798203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.709153 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:52.935501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.935463 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:52.936081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.936015 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:52.956001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.949514 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:52.968724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:52.968679 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:53.069441 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.069328 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:53.071406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.071374 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:54:53.072286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.072251 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:54:53.265645 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.265602 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:53.295091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.295026 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:54:53.349040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.349001 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:53.356581 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.356528 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:53.398283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.398250 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:54:53.492310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.492274 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:53.543040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.542999 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:54:53.546415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.546255 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:53.547633 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.544649 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:53.566030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.565983 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:54:53.567849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.567822 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:53.634657 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.634531 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:54:53.686713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.686677 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:54:53.761118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.757505 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:53.761118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.758952 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:54:53.762357 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.762297 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:53.964502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.957900 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:53.964502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.957967 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:53.964502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.958405 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:53.964502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:53.958435 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:54.152120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.127224 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:54:54.178335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.173167 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:54.199103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.182509 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:54:54.199103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.183522 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:54.320106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.319791 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:54.320106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.319851 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:54.416401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.416372 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:54:54.568590 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:54.567863 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.046587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.025189 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:54:55.046587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.025667 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.090635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.090287 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.090635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.090362 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:55.180538 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.180489 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.180867 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.180841 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.182603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.182575 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:54:55.188682 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.188128 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.198091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.195382 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.198091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.195431 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:55.237508 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.237464 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:55.301143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.299804 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" event={"ID":"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5","Type":"ContainerStarted","Data":"fc7601b658db4d08c642ba898482e5e27a1a0f1abef562aa04a9c4857509fbc5"} Oct 30 19:54:55.306972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.306918 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:54:55.309676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.309567 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.309676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.309619 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:55.345476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.345341 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" event={"ID":"ee5a5383-819d-4a37-9744-8dc81a4398a1","Type":"ContainerStarted","Data":"84716ea9dfbc3c898f22204392d2b2f296b0207eb8e0a068d09eeb01cd1b6665"} Oct 30 19:54:55.348088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.347475 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:54:55.374732 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.374703 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:54:55.397616 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.397434 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:55.398107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.397971 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:54:55.431167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.431077 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:54:55.439349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.439232 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:54:55.446369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.446282 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:54:55.607738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.601311 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:55.856215 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.856176 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:55.901398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.901341 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.901605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.901416 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 19:54:55.938786 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.938740 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 19:54:55.938963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:55.938825 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 19:54:56.304998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.304848 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.314787 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.314734 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 19:54:56.314996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.314805 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 19:54:56.443096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.441844 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" event={"ID":"ee5a5383-819d-4a37-9744-8dc81a4398a1","Type":"ContainerStarted","Data":"42db93ae19f1639e22e1405a311da6082fbb0a6ac8eac9c4cfb31be9ea11beaf"} Oct 30 19:54:56.443391 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.443321 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:54:56.447092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.446048 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" event={"ID":"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5","Type":"ContainerStarted","Data":"36452cdb29acd822220ff5bcc56854b012674b269bf14a2908f92fa67d41e451"} Oct 30 19:54:56.447092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.446982 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:54:56.567219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.539195 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:54:56.690227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.690163 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" podStartSLOduration=8.690142867 podStartE2EDuration="8.690142867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:56.524419419 +0000 UTC m=+1456.592254988" watchObservedRunningTime="2026-10-30 19:54:56.690142867 +0000 UTC m=+1456.757978438" Oct 30 19:54:56.692438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.692376 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.731124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.720717 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:54:56.731124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.723423 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d07cb404-openshift-default-dockercfg-fsqsp\"" Oct 30 19:54:56.731124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.725598 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" podStartSLOduration=8.725581289 podStartE2EDuration="8.725581289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:56.638234151 +0000 UTC m=+1456.706069716" watchObservedRunningTime="2026-10-30 19:54:56.725581289 +0000 UTC m=+1456.793416858" Oct 30 19:54:56.742892 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.742732 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893149 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893269 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893379 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893445 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fg85\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893772 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.893870 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.894611 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.894697 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:56.904015 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.894758 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999133 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999220 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fg85\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999336 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999400 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999424 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999489 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999555 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999668 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:56.999758 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.017423 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.022080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.018234 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.023072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.023028 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.024161 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.024076 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.033929 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.033872 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.034419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.034400 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.052115 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.041213 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.077438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.055946 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fg85\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.077438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.064653 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\") pod \"e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.368081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.367168 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:57.485077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.484049 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:57.502523 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.499516 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:57.502523 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.501124 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:57.502831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.502610 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 19:54:57.602091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.583132 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:54:57.773081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.770585 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:57.864948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.864899 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:57.864948 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.864954 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:54:57.865242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.865025 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:57.865242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.865154 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:57.870620 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.870594 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:57.870826 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.870812 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:57.870984 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.870970 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 19:54:57.873520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.873499 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 19:54:57.891115 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.889843 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ea210528\"" Oct 30 19:54:57.891115 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:57.889972 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ea210528-dockercfg-qp7x7\"" Oct 30 19:54:58.034881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.005649 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.034881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.005820 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslv4\" (UniqueName: \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.034881 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:58.019224 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca917a2_706d_45df_ac03_51ec5ae9df0b.slice/crio-04b4661bcfffb62986494e242b6e41971bfba360d0343022cab8231238be8ddd WatchSource:0}: Error finding container 04b4661bcfffb62986494e242b6e41971bfba360d0343022cab8231238be8ddd: Status 404 returned error can't find the container with id 04b4661bcfffb62986494e242b6e41971bfba360d0343022cab8231238be8ddd Oct 30 19:54:58.051455 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.051407 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:58.053467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.053430 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:58.053643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.053622 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:54:58.113076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.108653 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.113076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.110226 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslv4\" (UniqueName: \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.113076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.111563 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.168254 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.159544 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:54:58.228683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.228645 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslv4\" (UniqueName: \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\") pod \"payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.248078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.247442 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:58.271168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.270998 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:58.604432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.604399 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerStarted","Data":"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280"} Oct 30 19:54:58.604978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.604956 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerStarted","Data":"04b4661bcfffb62986494e242b6e41971bfba360d0343022cab8231238be8ddd"} Oct 30 19:54:58.647149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.647119 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:54:58.648011 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:54:58.647715 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ceb93_ec9d_4c76_8f5c_e3fdfb71dffa.slice/crio-b0b4a8595b5eba90375b9a780e2ec2f53756c9de4c3c862b84c9cc1c28e59983 WatchSource:0}: Error finding container b0b4a8595b5eba90375b9a780e2ec2f53756c9de4c3c862b84c9cc1c28e59983: Status 404 returned error can't find the container with id b0b4a8595b5eba90375b9a780e2ec2f53756c9de4c3c862b84c9cc1c28e59983 Oct 30 19:54:58.648228 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:58.648177 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" podStartSLOduration=2.648164945 podStartE2EDuration="2.648164945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:58.647992785 +0000 UTC m=+1458.715828356" watchObservedRunningTime="2026-10-30 19:54:58.648164945 +0000 UTC m=+1458.716000512" Oct 30 19:54:59.368416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.368387 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:59.375745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.375712 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:59.381364 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:54:59.381323 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:54:59.614819 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.614782 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" event={"ID":"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa","Type":"ContainerStarted","Data":"bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553"} Oct 30 19:54:59.615350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.614843 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" event={"ID":"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa","Type":"ContainerStarted","Data":"b0b4a8595b5eba90375b9a780e2ec2f53756c9de4c3c862b84c9cc1c28e59983"} Oct 30 19:54:59.615519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.615463 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:59.615817 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.615790 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:54:59.616978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.616917 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:54:59.665800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.665640 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" podStartSLOduration=2.665612025 podStartE2EDuration="2.665612025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:59.647728933 +0000 UTC m=+1459.715564515" watchObservedRunningTime="2026-10-30 19:54:59.665612025 +0000 UTC m=+1459.733447612" Oct 30 19:54:59.769700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:54:59.769637 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.422227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:00.422194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:00.705008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:00.704931 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.487957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.487906 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.654322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.654282 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.858415 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.858375 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:01.880667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.880629 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:55:01.886103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.886069 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:01.887721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.887690 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:55:01.892387 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.892362 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:01.895456 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.895428 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvlst\" (UniqueName: \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:01.895600 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.895566 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk7mh\" (UniqueName: \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:01.895694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.895673 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:01.895757 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.895730 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:01.897614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.897588 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-d07cb404\"" Oct 30 19:55:01.898010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.897989 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:55:01.902720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.902688 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-d07cb404-dockercfg-zrwkn\"" Oct 30 19:55:01.917290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:01.917259 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:55:02.076635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.076592 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvlst\" (UniqueName: \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:02.076828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.076701 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk7mh\" (UniqueName: \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:02.076828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.076801 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:02.076961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.076849 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:02.081079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.078027 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:02.229223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.228325 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.248216 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.242988 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:02.271076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.253521 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvlst\" (UniqueName: \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\") pod \"payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:02.274203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.271889 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk7mh\" (UniqueName: \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\") pod \"payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:02.513231 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.513125 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:02.522085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.521979 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:02.912906 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.912879 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:55:02.926470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:02.925662 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:55:03.207721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.207603 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.701496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.701456 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" event={"ID":"8689889f-1e71-49a8-82d9-a4629ae3eac8","Type":"ContainerStarted","Data":"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc"} Oct 30 19:55:03.701709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.701564 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:03.701709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.701585 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" event={"ID":"8689889f-1e71-49a8-82d9-a4629ae3eac8","Type":"ContainerStarted","Data":"339531ff0a6ff9418521e5240cc21d64bbfec522e4db6180ff3ddb2409c2cace"} Oct 30 19:55:03.703328 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.703300 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" event={"ID":"d9c1a314-4f52-42d1-9ba7-03135e4a526f","Type":"ContainerStarted","Data":"673a883f92efbe9273cca6a9a50f07890a0dff7bc5860dc5b8a1557ba366e7f6"} Oct 30 19:55:03.703434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.703346 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" event={"ID":"d9c1a314-4f52-42d1-9ba7-03135e4a526f","Type":"ContainerStarted","Data":"b9335d86c92c1f183754a764a0e4710da9841c8e77abc703a8d64e01032a322a"} Oct 30 19:55:03.703434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.703367 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:03.743673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.743620 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" podStartSLOduration=2.7436057419999997 podStartE2EDuration="2.743605742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:03.741287842 +0000 UTC m=+1463.809123413" watchObservedRunningTime="2026-10-30 19:55:03.743605742 +0000 UTC m=+1463.811441310" Oct 30 19:55:03.780749 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:03.780694 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" podStartSLOduration=2.7806820500000002 podStartE2EDuration="2.78068205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:03.772867501 +0000 UTC m=+1463.840703073" watchObservedRunningTime="2026-10-30 19:55:03.78068205 +0000 UTC m=+1463.848517617" Oct 30 19:55:04.725848 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:04.725800 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.850940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:05.850904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:06.920078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:06.911256 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.926985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:07.926944 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:08.608712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:08.608681 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:55:08.608884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:08.608752 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:55:09.274100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.274050 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:55:09.278962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.278942 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.294544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.294521 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ea210528-serving-cert\"" Oct 30 19:55:09.299257 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.299230 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:55:09.418663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.418628 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrnx\" (UniqueName: \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.418852 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.418681 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.418908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.418860 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.419026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.419008 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.454721 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.454683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.526367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.526256 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.526534 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.526410 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.526704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.526682 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrnx\" (UniqueName: \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.526861 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.526744 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.527841 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.527809 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.528459 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:09.528433 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-serving-cert: secret "maas-api-e2e-ait-ea210528-serving-cert" not found Oct 30 19:55:09.528577 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:09.528515 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls podName:4a4e5553-b95c-447d-a76a-fabbb7be36f9 nodeName:}" failed. No retries permitted until 2026-10-30 19:55:10.028496139 +0000 UTC m=+1470.096331700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls") pod "maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" (UID: "4a4e5553-b95c-447d-a76a-fabbb7be36f9") : secret "maas-api-e2e-ait-ea210528-serving-cert" not found Oct 30 19:55:09.534487 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.534462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.549690 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.549660 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrnx\" (UniqueName: \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:09.638295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.638251 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.784932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:09.784846 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.033899 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.033866 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:10.036667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.036557 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") pod \"maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:10.195406 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.195368 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:10.607186 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.607148 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:55:10.607610 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:55:10.607283 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a4e5553_b95c_447d_a76a_fabbb7be36f9.slice/crio-f92093bbb3265c400e2700ef2cd2c1a559693dfee4eec83745dc0669e86e5875 WatchSource:0}: Error finding container f92093bbb3265c400e2700ef2cd2c1a559693dfee4eec83745dc0669e86e5875: Status 404 returned error can't find the container with id f92093bbb3265c400e2700ef2cd2c1a559693dfee4eec83745dc0669e86e5875 Oct 30 19:55:10.662081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.660822 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 19:55:10.668491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.668461 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.689854 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.686220 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 19:55:10.777532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.777497 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789113 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789151 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789269 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789494 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789525 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbrvc\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789550 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789681 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789716 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.790158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.789764 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.795878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.794995 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" event={"ID":"4a4e5553-b95c-447d-a76a-fabbb7be36f9","Type":"ContainerStarted","Data":"f92093bbb3265c400e2700ef2cd2c1a559693dfee4eec83745dc0669e86e5875"} Oct 30 19:55:10.891134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891015 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891100 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891168 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891209 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891225 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891305 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891599 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891457 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891599 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891489 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbrvc\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891599 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891527 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891755 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891638 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891833 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891806 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.891833 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.891824 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.892126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.892104 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.892220 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.892159 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.893893 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.893869 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.893966 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.893871 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.905720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.905692 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.907114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.907089 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbrvc\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\") pod \"e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:10.988653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:10.988615 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:11.310862 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:55:11.310832 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a1ff27_b5f3_4816_abcf_446f2c4b87e6.slice/crio-f8348a7b09b0045dacfca9e9a44161ea94e741fd3da2c43c2f6b35bd3000971c WatchSource:0}: Error finding container f8348a7b09b0045dacfca9e9a44161ea94e741fd3da2c43c2f6b35bd3000971c: Status 404 returned error can't find the container with id f8348a7b09b0045dacfca9e9a44161ea94e741fd3da2c43c2f6b35bd3000971c Oct 30 19:55:11.312181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.312153 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 19:55:11.313941 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.313905 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:11.314010 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.313994 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:11.314069 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.314037 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:11.655295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.655268 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:55:11.830761 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.830144 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" event={"ID":"4a4e5553-b95c-447d-a76a-fabbb7be36f9","Type":"ContainerStarted","Data":"ac245dc33b4e13170f4ec2ad12d3d64a2f51d4e0d37572509d5d296fb065f084"} Oct 30 19:55:11.831754 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.831723 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:11.835470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.835434 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" event={"ID":"45a1ff27-b5f3-4816-abcf-446f2c4b87e6","Type":"ContainerStarted","Data":"71ba6b53b27a59815262e6cf7b3f14eb675f697471ee60857e1b3edf2605b157"} Oct 30 19:55:11.835636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.835495 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" event={"ID":"45a1ff27-b5f3-4816-abcf-446f2c4b87e6","Type":"ContainerStarted","Data":"f8348a7b09b0045dacfca9e9a44161ea94e741fd3da2c43c2f6b35bd3000971c"} Oct 30 19:55:11.862300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.862241 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" podStartSLOduration=2.493295908 podStartE2EDuration="2.862228287s" totalImagesPullingTime="368.932379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.860094948 +0000 UTC m=+1471.927930514" watchObservedRunningTime="2026-10-30 19:55:11.862228287 +0000 UTC m=+1471.930063857" Oct 30 19:55:11.892639 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.892589 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podStartSLOduration=1.89257636 podStartE2EDuration="1.89257636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:11.888932375 +0000 UTC m=+1471.956767946" watchObservedRunningTime="2026-10-30 19:55:11.89257636 +0000 UTC m=+1471.960411927" Oct 30 19:55:11.921463 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.921384 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:11.989683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:11.989605 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:12.990605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:12.990564 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:12.993067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:12.990626 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:13.148955 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:13.148922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.452412 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:13.452378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.995188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:13.995153 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.995803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:13.995767 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:14.736768 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:14.736737 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:55:14.739249 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:14.739220 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:55:14.995160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:14.994765 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:14.995160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:14.994819 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:15.080812 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:15.080764 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:15.995079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:15.994528 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:15.995079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:15.994592 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:17.114080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.113327 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:17.139270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.138863 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:17.139270 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.138934 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:17.428126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.427264 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:17.595142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.594905 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:17.611418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.596393 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 19:55:17.767399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:17.764645 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.091276 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.071614 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:18.091276 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.071684 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:18.120094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.115364 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.134090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.133140 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.335082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.300739 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.430419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.430356 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:18.600131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.599933 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.647081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.627758 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:18.756851 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.636163 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.789566 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.766236 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:18.806082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.774282 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:55:18.885688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.885639 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:18.902419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.902338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:18.996292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.996244 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:18.996591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:18.996544 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:19.165422 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.165368 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:55:19.187438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.187399 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:55:19.216957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.216918 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:19.244170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.244124 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.245428 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.245019 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:19.245641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.245616 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.247203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.247180 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.282583 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:19.282531 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.211s" Oct 30 19:55:19.282933 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.282915 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:55:19.312673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.312644 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:55:19.330474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.330443 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:55:19.331576 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.331553 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:55:19.341211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.341183 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:55:19.341492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.341474 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:55:19.398082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.372142 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.427479 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.400726 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:55:19.474131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.442931 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:19.605283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.605246 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.622221 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.622157 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.670807 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.668165 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.810197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.787635 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:19.810197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.793951 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:55:19.810197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.795423 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:19.810676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.810639 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:55:19.812969 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.812925 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:55:19.860081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.835225 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:19.860839 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.860809 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:55:20.056105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:19.989483 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:20.123112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.114133 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:20.221203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.182391 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.221203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.220228 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:55:20.340201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.340148 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 19:55:20.379157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.372438 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.421468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.421428 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:20.448950 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.448882 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.691147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.687944 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.822107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.715533 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:20.822107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.812340 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.825350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.825315 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:20.825621 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.825575 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.912382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.897211 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:20.913942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.897290 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:20.938084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.898457 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:20.938084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.922122 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.938084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.922266 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:20.947135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:20.943131 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.152088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.139699 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:21.152088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.139798 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:21.216099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.195214 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.220582 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.199464 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:21.220930 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.200250 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.396201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.396148 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.463362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.436598 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.463362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.438853 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:21.463362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.438892 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.477133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.475287 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.477133 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.475673 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.515770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.515727 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:55:21.539647 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:21.539615 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.477s" Oct 30 19:55:21.539911 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.539895 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:55:21.540164 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.540149 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:55:21.756616 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.756505 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Oct 30 19:55:21.908872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.908829 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:21.916193 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.916132 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:21.922942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.922903 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:21.974310 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.972482 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:21.989808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:21.985024 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:22.026703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.026615 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:55:22.027001 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.026985 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:55:22.028687 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.026745 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:22.028857 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.028836 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:22.338156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.309428 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:22.442591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.441002 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:22.442591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.441407 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:55:22.503679 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.502470 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:22.551839 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.542269 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:22.576659 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.569450 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Oct 30 19:55:22.594683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.593577 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:22.780079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.754351 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:55:22.898351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.898266 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:55:22.902922 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.902895 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:55:22.905704 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.905682 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:55:22.910333 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.910305 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 19:55:22.964717 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.964687 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 19:55:22.965158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.965137 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:55:22.965539 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.964995 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.965712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.965687 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:55:22.968800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.968779 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:55:22.968961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.968948 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:55:22.980141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.980112 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:55:22.980457 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.980438 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sh8sg\"" Oct 30 19:55:22.980856 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.980838 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:55:22.980971 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.980137 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:55:22.984264 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.984246 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:55:22.989006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.988982 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.989322 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.989299 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsccg\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.989530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.989509 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.989747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.989728 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.989946 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.989914 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.990241 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.990219 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.990485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.990464 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:22.991008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.990986 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:55:22.991193 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:22.991166 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Oct 30 19:55:23.094118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.093773 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.093881 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.094000 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsccg\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.094133 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.094241 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.094296 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.094625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.094523 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.095358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.095103 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.097683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.097551 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.098039 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.097989 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.098281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.098258 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.100207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.100178 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.123072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.123022 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.140075 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.139462 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsccg\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6h66z\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:23.306421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:23.306332 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:24.075975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.075879 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:24.076478 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.076332 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:24.080100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.079640 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 19:55:24.089151 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:55:24.089118 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7e5628_1d20_497b_a673_d28a54b125f2.slice/crio-bbb1dcfa4a9faccd0c08cceb5f23329831e3497ec437246a202784668c493c2c WatchSource:0}: Error finding container bbb1dcfa4a9faccd0c08cceb5f23329831e3497ec437246a202784668c493c2c: Status 404 returned error can't find the container with id bbb1dcfa4a9faccd0c08cceb5f23329831e3497ec437246a202784668c493c2c Oct 30 19:55:24.091337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.091313 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 19:55:24.230653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.230613 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.900425 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:24.900395 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:25.105975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:25.105938 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" event={"ID":"9a7e5628-1d20-497b-a673-d28a54b125f2","Type":"ContainerStarted","Data":"bbb1dcfa4a9faccd0c08cceb5f23329831e3497ec437246a202784668c493c2c"} Oct 30 19:55:25.232115 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:25.231977 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:26.307908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:26.307868 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:26.622006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:26.621911 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:27.498612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:27.498578 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:27.498928 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:27.498647 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:55:28.154173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.154135 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" event={"ID":"9a7e5628-1d20-497b-a673-d28a54b125f2","Type":"ContainerStarted","Data":"1e2a47b0561d9eea882ec56a468c8499de6b30d12716a739608e4ad974587968"} Oct 30 19:55:28.154589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.154558 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:28.156546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.156519 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6h66z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:28.156714 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.156579 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:28.198313 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.198252 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podStartSLOduration=2.792725856 podStartE2EDuration="6.198234812s" totalImagesPullingTime="3.405508956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:28.194651651 +0000 UTC m=+1488.262487219" watchObservedRunningTime="2026-10-30 19:55:28.198234812 +0000 UTC m=+1488.266070379" Oct 30 19:55:28.385733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:28.385685 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.164447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:29.164417 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 19:55:29.390649 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:29.390611 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:55:30.109821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:30.109785 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.163652 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:31.163614 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.732426 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:32.732385 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:34.327869 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:34.327832 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.451236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:35.451195 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.641377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:36.641338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.141350 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:37.141317 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.440366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:37.440288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.665385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:37.665338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.367980 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:38.367925 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.096253 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:39.096216 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:55:39.125675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:39.125645 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:55:39.131571 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:39.131540 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.285736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:39.285689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.477342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:39.477256 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.168108 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:40.167493 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.387243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:40.387192 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.569141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:40.566590 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:41.256051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:41.256007 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:41.700548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:41.700514 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.260185 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:42.256930 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.557835 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:42.557752 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.876669 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:42.876628 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:43.449119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:43.449023 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.097151 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.097106 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" podUID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" containerName="payload-processing" containerID="cri-o://36452cdb29acd822220ff5bcc56854b012674b269bf14a2908f92fa67d41e451" gracePeriod=25 Oct 30 19:55:44.126396 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.126345 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" podUID="ee5a5383-819d-4a37-9744-8dc81a4398a1" containerName="payload-pre-processing" containerID="cri-o://42db93ae19f1639e22e1405a311da6082fbb0a6ac8eac9c4cfb31be9ea11beaf" gracePeriod=25 Oct 30 19:55:44.334229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.333949 5733 generic.go:356] "Generic (PLEG): container finished" podID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" containerID="36452cdb29acd822220ff5bcc56854b012674b269bf14a2908f92fa67d41e451" exitCode=0 Oct 30 19:55:44.334229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.334143 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" event={"ID":"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5","Type":"ContainerDied","Data":"36452cdb29acd822220ff5bcc56854b012674b269bf14a2908f92fa67d41e451"} Oct 30 19:55:44.337044 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.336590 5733 generic.go:356] "Generic (PLEG): container finished" podID="ee5a5383-819d-4a37-9744-8dc81a4398a1" containerID="42db93ae19f1639e22e1405a311da6082fbb0a6ac8eac9c4cfb31be9ea11beaf" exitCode=0 Oct 30 19:55:44.337044 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.336657 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" event={"ID":"ee5a5383-819d-4a37-9744-8dc81a4398a1","Type":"ContainerDied","Data":"42db93ae19f1639e22e1405a311da6082fbb0a6ac8eac9c4cfb31be9ea11beaf"} Oct 30 19:55:44.463559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.463532 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:55:44.490516 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.490494 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.532482 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\") pod \"ee5a5383-819d-4a37-9744-8dc81a4398a1\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.532581 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\" (UniqueName: \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\") pod \"ee5a5383-819d-4a37-9744-8dc81a4398a1\" (UID: \"ee5a5383-819d-4a37-9744-8dc81a4398a1\") " Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.532662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\" (UniqueName: \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\") pod \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.532787 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\") pod \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\" (UID: \"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5\") " Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.533930 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume" pod "ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" (UID: "ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:44.535080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.534834 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume" pod "ee5a5383-819d-4a37-9744-8dc81a4398a1" (UID: "ee5a5383-819d-4a37-9744-8dc81a4398a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:44.538067 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.538030 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs" pod "ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" (UID: "ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5"). InnerVolumeSpecName "kube-api-access-28zqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:44.541473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.541445 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7" pod "ee5a5383-819d-4a37-9744-8dc81a4398a1" (UID: "ee5a5383-819d-4a37-9744-8dc81a4398a1"). InnerVolumeSpecName "kube-api-access-x62s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:44.584779 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.584688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.635785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.635697 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:44.635785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.635729 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5a5383-819d-4a37-9744-8dc81a4398a1-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:44.635785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.635743 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62s7\" (UniqueName: \"kubernetes.io/projected/ee5a5383-819d-4a37-9744-8dc81a4398a1-kube-api-access-x62s7\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:44.635785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:44.635756 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zqs\" (UniqueName: \"kubernetes.io/projected/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5-kube-api-access-28zqs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:45.346693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.346661 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" Oct 30 19:55:45.346693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.346682 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78" event={"ID":"ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5","Type":"ContainerDied","Data":"fc7601b658db4d08c642ba898482e5e27a1a0f1abef562aa04a9c4857509fbc5"} Oct 30 19:55:45.346935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.346746 5733 scope.go:122] "RemoveContainer" containerID="36452cdb29acd822220ff5bcc56854b012674b269bf14a2908f92fa67d41e451" Oct 30 19:55:45.348662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.348638 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" Oct 30 19:55:45.348762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.348666 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n" event={"ID":"ee5a5383-819d-4a37-9744-8dc81a4398a1","Type":"ContainerDied","Data":"84716ea9dfbc3c898f22204392d2b2f296b0207eb8e0a068d09eeb01cd1b6665"} Oct 30 19:55:45.361589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.361566 5733 scope.go:122] "RemoveContainer" containerID="42db93ae19f1639e22e1405a311da6082fbb0a6ac8eac9c4cfb31be9ea11beaf" Oct 30 19:55:45.397260 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.397237 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:55:45.404211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.404189 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-df345a65-6bbddfd57f-wtx9n"] Oct 30 19:55:45.426394 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.426370 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:55:45.432043 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.432019 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.432226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.432208 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-df345a65-6d656dcb86-dxg78"] Oct 30 19:55:45.785800 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.785665 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.988616 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:45.988515 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:46.056106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:46.055999 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" path="/var/lib/kubelet/pods/ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5/volumes" Oct 30 19:55:46.056769 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:46.056747 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5a5383-819d-4a37-9744-8dc81a4398a1" path="/var/lib/kubelet/pods/ee5a5383-819d-4a37-9744-8dc81a4398a1/volumes" Oct 30 19:55:46.409259 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:46.409227 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:46.570373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:46.570320 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:47.503589 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:47.503547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:47.547098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:47.547044 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:55:47.675544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:47.675493 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:47.824100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:47.823968 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.239515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:48.239476 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.408746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:48.408711 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.839468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:48.839428 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:49.953087 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:49.953038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.484945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:51.484902 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.699607 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:51.699572 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:51.949160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:51.949112 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:52.277679 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.277568 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.348014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.347973 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:55:52.451081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.450139 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.451293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.451198 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.472209 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.472157 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.493083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.492312 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.501082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.500101 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:55:52.510173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.510133 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.555407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.555135 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" podUID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" containerName="payload-processing" containerID="cri-o://bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553" gracePeriod=25 Oct 30 19:55:52.645083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.631010 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:55:52.696098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.670426 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.707290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.706826 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.990100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:52.989578 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:55:53.023247 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.023194 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:53.032080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.024899 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:55:53.212083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.047431 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:53.369486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.047476 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:53.519544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.519473 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:53.632136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.541142 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:55:53.632136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.613179 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:55:53.632136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.613251 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:55:53.741118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.187772 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:53.741118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.740448 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:53.757090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.741523 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:55:53.831268 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.824008 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:55:53.851287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.851235 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:55:54.011013 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.851488 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.871252 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.014328 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.014383 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:53.922828 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.023897 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.035883 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:54.037202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.036640 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:55:54.870138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:54.860250 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:55.063126 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.063079 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:55.092886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.092827 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:55.242296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.242162 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:55:55.283105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.246447 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:55.501152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.499166 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:55.501152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.499600 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 19:55:55.501152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.499919 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:55:55.501152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.500234 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:55.550194 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.531125 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 19:55:55.550194 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.531433 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 19:55:55.588084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.582419 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:55.965143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.905473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:55.992082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.967007 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 19:55:56.016083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:55.994575 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 30 19:55:56.097683 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.097638 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:56.106079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.105553 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:56.142143 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:56.119804 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.066s" Oct 30 19:55:56.158114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.153304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.239040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.238902 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.273200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.273151 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.275825 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.275778 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.383337 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.376419 5733 generic.go:356] "Generic (PLEG): container finished" podID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" containerID="bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553" exitCode=0 Oct 30 19:55:56.450283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.450250 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.505150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.504282 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.505150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.504698 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.516921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.505774 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.516921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.506046 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:56.533584 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.533539 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:56.965114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.956283 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 19:55:56.988543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:56.975583 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" event={"ID":"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa","Type":"ContainerDied","Data":"bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553"} Oct 30 19:55:57.002707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.002676 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:55:57.002965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.002948 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:55:57.042086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.031796 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.127081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.117458 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.143176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.131416 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:57.323079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.290355 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.336414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.310932 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.351438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.310968 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:57.509222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.368562 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:55:57.509222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.369364 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" podUID="8689889f-1e71-49a8-82d9-a4629ae3eac8" containerName="payload-processing" containerID="cri-o://0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc" gracePeriod=25 Oct 30 19:55:57.552771 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.552731 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" podUID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" containerName="payload-pre-processing" containerID="cri-o://673a883f92efbe9273cca6a9a50f07890a0dff7bc5860dc5b8a1557ba366e7f6" gracePeriod=25 Oct 30 19:55:57.622462 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.618505 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.796707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.751832 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.796707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.751912 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.796707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.772269 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:55:57.796707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.782857 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:57.796707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.783105 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 19:55:57.825091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:57.811717 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:58.048302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.048188 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:58.190082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.153778 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:58.298656 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.298558 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:58.497432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.497372 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 19:55:58.499079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.499017 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 19:55:58.515398 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.515363 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:55:58.762083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.754525 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:58.762083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.756251 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:55:58.787125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.787087 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:58.792447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.792395 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:58.808072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.808020 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:58.827843 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:58.827731 5733 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743ceb93_ec9d_4c76_8f5c_e3fdfb71dffa.slice/crio-conmon-bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:55:58.911225 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.911190 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 19:55:58.967092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:58.966524 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\") pod \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " Oct 30 19:55:59.000384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.000332 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:59.043094 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.031229 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:59.043414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.035702 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume" pod "743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" (UID: "743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:59.125083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.119418 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:59.179712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.179672 5733 generic.go:356] "Generic (PLEG): container finished" podID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" containerID="673a883f92efbe9273cca6a9a50f07890a0dff7bc5860dc5b8a1557ba366e7f6" exitCode=0 Oct 30 19:55:59.180433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.180405 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" event={"ID":"d9c1a314-4f52-42d1-9ba7-03135e4a526f","Type":"ContainerDied","Data":"673a883f92efbe9273cca6a9a50f07890a0dff7bc5860dc5b8a1557ba366e7f6"} Oct 30 19:55:59.348107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.348074 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 19:55:59.370223 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.368122 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 19:55:59.398258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.368381 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 19:55:59.405079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.401961 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 19:55:59.405079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.402670 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:59.467417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.467385 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 19:55:59.483703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.483670 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\" (UniqueName: \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\") pod \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\" (UID: \"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa\") " Oct 30 19:55:59.485341 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.485314 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.501086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.487649 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 19:55:59.501086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.497436 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 19:55:59.509073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.505869 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 19:55:59.509073 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.507483 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 19:55:59.530444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.530407 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:59.599625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.599542 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 19:55:59.689741 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.689658 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4" pod "743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" (UID: "743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa"). InnerVolumeSpecName "kube-api-access-mslv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:59.789102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.788995 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mslv4\" (UniqueName: \"kubernetes.io/projected/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa-kube-api-access-mslv4\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.797449 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:55:59.790659 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:55:59.797449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.790881 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:59.797449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.795637 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 19:55:59.797449 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:55:59.795743 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 19:56:00.195411 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.195380 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:56:00.201188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.198215 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:56:00.201188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.198664 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" Oct 30 19:56:00.201188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.199480 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9" event={"ID":"743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa","Type":"ContainerDied","Data":"b0b4a8595b5eba90375b9a780e2ec2f53756c9de4c3c862b84c9cc1c28e59983"} Oct 30 19:56:00.201188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.199553 5733 scope.go:122] "RemoveContainer" containerID="bc336a6286c93e5c0392fa13f2b78cba4c16f72e7ae42b03190622583ddf7553" Oct 30 19:56:00.204528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.204261 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" event={"ID":"d9c1a314-4f52-42d1-9ba7-03135e4a526f","Type":"ContainerDied","Data":"b9335d86c92c1f183754a764a0e4710da9841c8e77abc703a8d64e01032a322a"} Oct 30 19:56:00.204528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.204355 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95" Oct 30 19:56:00.206588 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.206562 5733 generic.go:356] "Generic (PLEG): container finished" podID="8689889f-1e71-49a8-82d9-a4629ae3eac8" containerID="0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc" exitCode=0 Oct 30 19:56:00.254291 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.254257 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" Oct 30 19:56:00.259437 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.254554 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" event={"ID":"8689889f-1e71-49a8-82d9-a4629ae3eac8","Type":"ContainerDied","Data":"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc"} Oct 30 19:56:00.259694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.259661 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb" event={"ID":"8689889f-1e71-49a8-82d9-a4629ae3eac8","Type":"ContainerDied","Data":"339531ff0a6ff9418521e5240cc21d64bbfec522e4db6180ff3ddb2409c2cace"} Oct 30 19:56:00.259834 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.256549 5733 scope.go:122] "RemoveContainer" containerID="673a883f92efbe9273cca6a9a50f07890a0dff7bc5860dc5b8a1557ba366e7f6" Oct 30 19:56:00.282390 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.282369 5733 scope.go:122] "RemoveContainer" containerID="0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc" Oct 30 19:56:00.299316 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.299289 5733 scope.go:122] "RemoveContainer" containerID="0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc" Oct 30 19:56:00.299940 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:56:00.299921 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc\": container with ID starting with 0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc not found: ID does not exist" containerID="0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc" Oct 30 19:56:00.300085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.300051 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc"} err="failed to get container status \"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc\": rpc error: code = NotFound desc = could not find container \"0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc\": container with ID starting with 0568423eedd7f31953534ec99776fe618eaf894f5419f0bb04622600d8876ffc not found: ID does not exist" Oct 30 19:56:00.306438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.306419 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\" (UniqueName: \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\") pod \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " Oct 30 19:56:00.306640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.306620 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\") pod \"8689889f-1e71-49a8-82d9-a4629ae3eac8\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " Oct 30 19:56:00.306813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.306800 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\") pod \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\" (UID: \"d9c1a314-4f52-42d1-9ba7-03135e4a526f\") " Oct 30 19:56:00.307600 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.307195 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume" pod "8689889f-1e71-49a8-82d9-a4629ae3eac8" (UID: "8689889f-1e71-49a8-82d9-a4629ae3eac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:00.307716 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.307618 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume" pod "d9c1a314-4f52-42d1-9ba7-03135e4a526f" (UID: "d9c1a314-4f52-42d1-9ba7-03135e4a526f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:00.308192 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.306948 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\" (UniqueName: \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\") pod \"8689889f-1e71-49a8-82d9-a4629ae3eac8\" (UID: \"8689889f-1e71-49a8-82d9-a4629ae3eac8\") " Oct 30 19:56:00.309516 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.309493 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c1a314-4f52-42d1-9ba7-03135e4a526f-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:00.309646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.309631 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8689889f-1e71-49a8-82d9-a4629ae3eac8-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:00.352356 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.350593 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst" pod "8689889f-1e71-49a8-82d9-a4629ae3eac8" (UID: "8689889f-1e71-49a8-82d9-a4629ae3eac8"). InnerVolumeSpecName "kube-api-access-bvlst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:00.352356 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.351875 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh" pod "d9c1a314-4f52-42d1-9ba7-03135e4a526f" (UID: "d9c1a314-4f52-42d1-9ba7-03135e4a526f"). InnerVolumeSpecName "kube-api-access-dk7mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:00.375125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.363785 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:56:00.382997 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.382587 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ea210528-5b6b9f9758-mjkv9"] Oct 30 19:56:00.451210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.448635 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:00.451210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.450464 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk7mh\" (UniqueName: \"kubernetes.io/projected/d9c1a314-4f52-42d1-9ba7-03135e4a526f-kube-api-access-dk7mh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:00.451210 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.450491 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvlst\" (UniqueName: \"kubernetes.io/projected/8689889f-1e71-49a8-82d9-a4629ae3eac8-kube-api-access-bvlst\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:00.671007 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.670942 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:56:00.679187 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.679156 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-d07cb404-6c89778d9f-f5s95"] Oct 30 19:56:00.688118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.686564 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:56:00.694713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:00.694682 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-d07cb404-7ddf8688b8-8lhwb"] Oct 30 19:56:01.593334 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:01.593299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:02.054114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:02.054075 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" path="/var/lib/kubelet/pods/743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa/volumes" Oct 30 19:56:02.054624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:02.054607 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8689889f-1e71-49a8-82d9-a4629ae3eac8" path="/var/lib/kubelet/pods/8689889f-1e71-49a8-82d9-a4629ae3eac8/volumes" Oct 30 19:56:02.055138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:02.055117 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" path="/var/lib/kubelet/pods/d9c1a314-4f52-42d1-9ba7-03135e4a526f/volumes" Oct 30 19:56:03.110089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:03.099599 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.338168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:03.338118 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.603975 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:03.603942 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.926110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:04.922669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.175860 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:05.175679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:06.252593 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:06.252558 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:06.940849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:06.940810 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:56:06.957577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:06.941087 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" podUID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" containerName="maas-api" containerID="cri-o://ac245dc33b4e13170f4ec2ad12d3d64a2f51d4e0d37572509d5d296fb065f084" gracePeriod=30 Oct 30 19:56:07.067506 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.067474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.317137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.316534 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.445532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.445499 5733 generic.go:356] "Generic (PLEG): container finished" podID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerID="da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280" exitCode=0 Oct 30 19:56:07.445671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.445574 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerDied","Data":"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280"} Oct 30 19:56:07.446348 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.446159 5733 scope.go:122] "RemoveContainer" containerID="da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280" Oct 30 19:56:07.447954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.447923 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:07.448020 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.447997 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:07.448085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.448031 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:56:07.448903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.448806 5733 generic.go:356] "Generic (PLEG): container finished" podID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" containerID="ac245dc33b4e13170f4ec2ad12d3d64a2f51d4e0d37572509d5d296fb065f084" exitCode=0 Oct 30 19:56:07.448903 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.448863 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" event={"ID":"4a4e5553-b95c-447d-a76a-fabbb7be36f9","Type":"ContainerDied","Data":"ac245dc33b4e13170f4ec2ad12d3d64a2f51d4e0d37572509d5d296fb065f084"} Oct 30 19:56:07.472351 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.472322 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:56:07.586501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.585408 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\") pod \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " Oct 30 19:56:07.586501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.585457 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") pod \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " Oct 30 19:56:07.586501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.585713 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\") pod \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " Oct 30 19:56:07.586501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.585897 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\" (UniqueName: \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\") pod \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\" (UID: \"4a4e5553-b95c-447d-a76a-fabbb7be36f9\") " Oct 30 19:56:07.589000 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.588972 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca" pod "4a4e5553-b95c-447d-a76a-fabbb7be36f9" (UID: "4a4e5553-b95c-447d-a76a-fabbb7be36f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:07.589345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.589257 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls" pod "4a4e5553-b95c-447d-a76a-fabbb7be36f9" (UID: "4a4e5553-b95c-447d-a76a-fabbb7be36f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:07.590491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.590472 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs" pod "4a4e5553-b95c-447d-a76a-fabbb7be36f9" (UID: "4a4e5553-b95c-447d-a76a-fabbb7be36f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:07.590945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.590916 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx" pod "4a4e5553-b95c-447d-a76a-fabbb7be36f9" (UID: "4a4e5553-b95c-447d-a76a-fabbb7be36f9"). InnerVolumeSpecName "kube-api-access-csrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:07.689018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.688985 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csrnx\" (UniqueName: \"kubernetes.io/projected/4a4e5553-b95c-447d-a76a-fabbb7be36f9-kube-api-access-csrnx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.689018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.689015 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a4e5553-b95c-447d-a76a-fabbb7be36f9-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.689018 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.689026 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.689256 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.689035 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a4e5553-b95c-447d-a76a-fabbb7be36f9-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.816685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:07.816595 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.017959 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.017921 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.460949 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.460912 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerStarted","Data":"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8"} Oct 30 19:56:08.462484 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.462454 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" event={"ID":"4a4e5553-b95c-447d-a76a-fabbb7be36f9","Type":"ContainerDied","Data":"f92093bbb3265c400e2700ef2cd2c1a559693dfee4eec83745dc0669e86e5875"} Oct 30 19:56:08.462602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.462491 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s" Oct 30 19:56:08.462602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.462518 5733 scope.go:122] "RemoveContainer" containerID="ac245dc33b4e13170f4ec2ad12d3d64a2f51d4e0d37572509d5d296fb065f084" Oct 30 19:56:08.557537 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.557492 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:56:08.562366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:08.562337 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ea210528-6cdbf97ddc-7wq5s"] Oct 30 19:56:09.096419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:09.096381 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:09.370346 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:09.370256 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:56:09.370520 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:09.370489 5733 patch_prober.go:29] interesting pod/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:56:09.370585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:09.370538 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 19:56:09.443305 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:09.443262 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.001214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.001170 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.060482 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.060446 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" path="/var/lib/kubelet/pods/4a4e5553-b95c-447d-a76a-fabbb7be36f9/volumes" Oct 30 19:56:10.152470 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.152421 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.370141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.369484 5733 patch_prober.go:29] interesting pod/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 19:56:10.370141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.369559 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 19:56:10.441123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.441088 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.681227 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.677676 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.848079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.847134 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:10.860102 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.858923 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:56:10.868214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:10.862931 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" containerID="cri-o://fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8" gracePeriod=30 Oct 30 19:56:11.077136 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:11.076907 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:11.903978 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:11.903931 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.142255 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:12.142221 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.305891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:12.305795 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.902045 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:12.902004 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.976796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:13.976760 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.161765 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:14.161726 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.499702 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:14.499660 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.615471 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:14.615441 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:56:15.134598 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:15.134567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.480197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:15.463983 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.640225 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:15.640142 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.487399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.487358 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 19:56:16.488935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488897 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" containerName="payload-processing" Oct 30 19:56:16.488935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488925 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488953 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ee5a5383-819d-4a37-9744-8dc81a4398a1" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488961 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5a5383-819d-4a37-9744-8dc81a4398a1" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488986 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" containerName="maas-api" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.488994 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" containerName="maas-api" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489037 5733 cpu_manager.go:414] "containerMap: removing container" podUID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489045 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489073 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489081 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489091 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8689889f-1e71-49a8-82d9-a4629ae3eac8" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489098 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8689889f-1e71-49a8-82d9-a4629ae3eac8" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489322 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c1a314-4f52-42d1-9ba7-03135e4a526f" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489344 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8689889f-1e71-49a8-82d9-a4629ae3eac8" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489369 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6d01f1-c8d1-4322-8b5c-e56f5c9444b5" containerName="payload-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489383 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5a5383-819d-4a37-9744-8dc81a4398a1" containerName="payload-pre-processing" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489398 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4e5553-b95c-447d-a76a-fabbb7be36f9" containerName="maas-api" Oct 30 19:56:16.489530 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.489430 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="743ceb93-ec9d-4c76-8f5c-e3fdfb71dffa" containerName="payload-processing" Oct 30 19:56:16.498478 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.495781 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.505501 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.505477 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a8de3b-serving-cert\"" Oct 30 19:56:16.507399 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.507373 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 19:56:16.571878 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.571848 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.644478 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.644428 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.644689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.644495 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pb9\" (UniqueName: \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.644882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.644842 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.644981 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.644932 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.737476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.737383 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.746146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.746104 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.746326 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.746248 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.746326 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.746303 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pb9\" (UniqueName: \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.747694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.746705 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.747694 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.747629 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.749500 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.749389 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.765303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.765263 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pb9\" (UniqueName: \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.773626 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.773594 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.822973 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.822938 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:16.913047 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:16.898756 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:17.126229 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:56:17.126198 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfc13ba_74ce_4e27_af6f_17b6c1a24c18.slice/crio-280df8ceef2d4a5484429d18f55006123a475f859b749c82bd1843496ce65d60 WatchSource:0}: Error finding container 280df8ceef2d4a5484429d18f55006123a475f859b749c82bd1843496ce65d60: Status 404 returned error can't find the container with id 280df8ceef2d4a5484429d18f55006123a475f859b749c82bd1843496ce65d60 Oct 30 19:56:17.127936 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:17.127905 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 19:56:17.706160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:17.706123 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerStarted","Data":"280df8ceef2d4a5484429d18f55006123a475f859b749c82bd1843496ce65d60"} Oct 30 19:56:17.826088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:17.825749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.644514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:18.643532 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.718865 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:18.718825 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerStarted","Data":"860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307"} Oct 30 19:56:18.719378 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:18.718965 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:18.752891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:18.752796 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podStartSLOduration=2.271500932 podStartE2EDuration="2.752774326s" totalImagesPullingTime="481.273394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:18.746090527 +0000 UTC m=+1538.813926094" watchObservedRunningTime="2026-10-30 19:56:18.752774326 +0000 UTC m=+1538.820609895" Oct 30 19:56:19.534570 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.534539 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:56:19.629763 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.629716 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.629977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.629816 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.629977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.629861 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.629977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.629892 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.629977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.629926 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.630258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630028 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.630258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630124 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.630258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630200 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.630425 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630375 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:19.630425 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630387 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\") pod \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\" (UID: \"7ca917a2-706d-45df-ac03-51ec5ae9df0b\") " Oct 30 19:56:19.630636 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630613 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:19.630918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630896 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:19.631014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.630920 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:19.631252 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631228 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:19.631468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631430 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.631468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631453 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.631468 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631466 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.631667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631479 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.631667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.631493 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.633833 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.633802 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:19.633952 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.633851 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "kube-api-access-9fg85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:19.634168 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.634144 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:19.634345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.634321 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy" pod "7ca917a2-706d-45df-ac03-51ec5ae9df0b" (UID: "7ca917a2-706d-45df-ac03-51ec5ae9df0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:19.730519 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.730483 5733 generic.go:356] "Generic (PLEG): container finished" podID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerID="fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8" exitCode=0 Oct 30 19:56:19.730931 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.730533 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" Oct 30 19:56:19.730931 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.730571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerDied","Data":"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8"} Oct 30 19:56:19.730931 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.730639 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5" event={"ID":"7ca917a2-706d-45df-ac03-51ec5ae9df0b","Type":"ContainerDied","Data":"04b4661bcfffb62986494e242b6e41971bfba360d0343022cab8231238be8ddd"} Oct 30 19:56:19.730931 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.730674 5733 scope.go:122] "RemoveContainer" containerID="fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8" Oct 30 19:56:19.732750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.732720 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fg85\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-kube-api-access-9fg85\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.732750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.732744 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.732750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.732754 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.732994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.732763 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca917a2-706d-45df-ac03-51ec5ae9df0b-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:19.744208 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.744188 5733 scope.go:122] "RemoveContainer" containerID="da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280" Oct 30 19:56:19.754844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.754816 5733 scope.go:122] "RemoveContainer" containerID="fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8" Oct 30 19:56:19.755154 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:56:19.755122 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8\": container with ID starting with fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8 not found: ID does not exist" containerID="fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8" Oct 30 19:56:19.755267 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.755163 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8"} err="failed to get container status \"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8\": rpc error: code = NotFound desc = could not find container \"fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8\": container with ID starting with fdd07904bc981e9d47719806de98c3d5ec18e9402f65145bcb59f523d2067cd8 not found: ID does not exist" Oct 30 19:56:19.755267 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.755187 5733 scope.go:122] "RemoveContainer" containerID="da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280" Oct 30 19:56:19.755485 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:56:19.755463 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280\": container with ID starting with da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280 not found: ID does not exist" containerID="da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280" Oct 30 19:56:19.755557 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.755497 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280"} err="failed to get container status \"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280\": rpc error: code = NotFound desc = could not find container \"da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280\": container with ID starting with da06b7dc6ac771435220131a1f2c0fdaf1bcd0d7b7cca0c5fa343b872d89a280 not found: ID does not exist" Oct 30 19:56:19.766475 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.766445 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:56:19.771202 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:19.771178 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d07cb404-openshift-default-77f85c89c7-b4wn5"] Oct 30 19:56:20.003122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:20.002955 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.059441 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:20.059397 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" path="/var/lib/kubelet/pods/7ca917a2-706d-45df-ac03-51ec5ae9df0b/volumes" Oct 30 19:56:20.863446 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:20.863291 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.970821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:21.969426 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.797281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:22.797183 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.499147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.499109 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:56:23.500248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.500209 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:56:23.500248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.500236 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:56:23.500454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.500273 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:56:23.500454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.500282 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:56:23.500564 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.500471 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:56:23.505238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.505215 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.526366 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.526335 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e7f2a520-serving-cert\"" Oct 30 19:56:23.532416 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.532387 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:56:23.594188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.594153 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.598971 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.598942 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg45r\" (UniqueName: \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.599139 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.599123 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.599244 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.599223 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.599290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.599281 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.700863 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.700818 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.701972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.701945 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.702208 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.702189 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.702584 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.702566 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg45r\" (UniqueName: \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.702695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.702669 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.706744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.706714 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.708369 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.708223 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.722423 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.722391 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg45r\" (UniqueName: \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\") pod \"maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:23.767358 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.767266 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.882859 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:23.882827 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:24.193783 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.193749 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:56:24.738138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.738103 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 19:56:24.794124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.790790 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" event={"ID":"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b","Type":"ContainerStarted","Data":"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995"} Oct 30 19:56:24.794124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.790857 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" event={"ID":"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b","Type":"ContainerStarted","Data":"edc138d529b6157826fed9977f3482ba3582d5cb7063f35dcc6109e390d617e3"} Oct 30 19:56:24.794124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.792141 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:24.802117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.799294 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.834079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:24.817992 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" podStartSLOduration=1.402416262 podStartE2EDuration="1.817963124s" totalImagesPullingTime="415.546862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:24.816360307 +0000 UTC m=+1544.884195878" watchObservedRunningTime="2026-10-30 19:56:24.817963124 +0000 UTC m=+1544.885798692" Oct 30 19:56:25.723079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:25.718549 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:26.587638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:26.587516 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:27.329659 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:27.329618 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:28.730236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:28.728130 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:29.629087 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:56:29.625716 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:56:30.500081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:30.499184 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 19:56:30.612081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:30.603283 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 19:56:30.812130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:30.811521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:30.868591 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:30.868169 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:31.030078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:31.029162 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:31.096137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:31.092126 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:31.829933 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:31.829899 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:56:32.138076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:32.137527 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.361116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:32.348689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.129081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:33.118690 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.512143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:33.504938 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:33.565622 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:33.553169 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:33.565622 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:33.554285 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:33.900469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:33.900437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.401490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:34.401459 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.939962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:34.939921 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:35.165276 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:35.165243 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.153909 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:36.153874 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:36.568454 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:36.568337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.082796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:37.082761 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c"] Oct 30 19:56:37.089777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:37.089751 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bc9b3d5-gq84c"] Oct 30 19:56:37.414845 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:37.414809 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.067919 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:38.067884 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e985d03f-0643-472c-ab7f-d0cb147a2636" path="/var/lib/kubelet/pods/e985d03f-0643-472c-ab7f-d0cb147a2636/volumes" Oct 30 19:56:38.485393 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:38.485359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:39.916137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:39.916038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:40.027666 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:40.027630 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf"] Oct 30 19:56:40.036954 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:40.036923 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e2166da-q5vrf"] Oct 30 19:56:40.055549 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:40.055519 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcccf22c-5864-408d-b806-0dd13fba3cff" path="/var/lib/kubelet/pods/fcccf22c-5864-408d-b806-0dd13fba3cff/volumes" Oct 30 19:56:40.874104 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:40.873425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.137211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:41.137099 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.342335 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:41.342278 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:41.592235 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:41.592155 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.089397 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:42.089360 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.497339 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:42.497309 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:42.696700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:42.696666 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.603481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:43.603429 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:44.441419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:44.441383 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:44.957880 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:44.957843 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.152436 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:45.152395 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.394117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:45.373974 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.418092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:46.417756 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:47.214573 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:47.214499 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.094325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:48.094286 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.506827 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:48.506749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.846802 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:48.846763 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.045731 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:49.045694 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv"] Oct 30 19:56:49.050568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:49.050540 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e56dbeb5-k69vv"] Oct 30 19:56:49.745791 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:49.745752 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.054372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:50.054284 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e86dcc-ee0a-49c8-85f3-4c984fd07004" path="/var/lib/kubelet/pods/f3e86dcc-ee0a-49c8-85f3-4c984fd07004/volumes" Oct 30 19:56:50.688128 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:50.688088 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.624608 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:51.624574 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.811671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:51.811635 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:52.385785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:52.385748 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:52.895490 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:52.895423 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.801248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:53.801211 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.716587 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:54.714665 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.474229 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:55.474177 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.618655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:55.618616 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.775994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:55.775903 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.669832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:56.669777 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:57.134571 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:57.134530 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm"] Oct 30 19:56:57.162942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:57.162909 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28dac7f8-88vmm"] Oct 30 19:56:57.920762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:57.920713 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:58.051621 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:58.051586 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="341bb8d5-89c5-44b2-9a6b-0e97287134f7" path="/var/lib/kubelet/pods/341bb8d5-89c5-44b2-9a6b-0e97287134f7/volumes" Oct 30 19:56:58.677336 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:58.677298 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:59.522079 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:56:59.517155 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:56:59.837929 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:56:59.837844 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.128811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.128758 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:57:01.129596 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.129562 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerName="maas-api" containerID="cri-o://06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995" gracePeriod=30 Oct 30 19:57:01.232414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.232372 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.421745 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:57:01.421659 5733 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fcca0f_e02f_4881_89f6_16caf9ae4a5b.slice/crio-06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:57:01.729361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.700377 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.884868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.884812 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:01.940432 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.940278 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:57:01.953091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.949651 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\") pod \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " Oct 30 19:57:01.953091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.949883 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\") pod \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " Oct 30 19:57:01.953091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.950137 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\") pod \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " Oct 30 19:57:01.953091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.950190 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\" (UniqueName: \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\") pod \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\" (UID: \"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b\") " Oct 30 19:57:01.962352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.961490 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca" pod "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" (UID: "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:01.962541 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.962480 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:01.993434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.993336 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r" pod "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" (UID: "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b"). InnerVolumeSpecName "kube-api-access-zg45r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:01.993608 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.993485 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls" pod "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" (UID: "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:01.993872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:01.993845 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs" pod "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" (UID: "d3fcca0f-e02f-4881-89f6-16caf9ae4a5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:02.064756 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.064717 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.064756 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.064754 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.065030 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.064771 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg45r\" (UniqueName: \"kubernetes.io/projected/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b-kube-api-access-zg45r\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:02.271797 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.269532 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.297086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.295984 5733 generic.go:356] "Generic (PLEG): container finished" podID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerID="06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995" exitCode=0 Oct 30 19:57:02.297086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.296051 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" event={"ID":"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b","Type":"ContainerDied","Data":"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995"} Oct 30 19:57:02.297086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.296124 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" event={"ID":"d3fcca0f-e02f-4881-89f6-16caf9ae4a5b","Type":"ContainerDied","Data":"edc138d529b6157826fed9977f3482ba3582d5cb7063f35dcc6109e390d617e3"} Oct 30 19:57:02.297086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.296154 5733 scope.go:122] "RemoveContainer" containerID="06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995" Oct 30 19:57:02.297086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.296368 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" Oct 30 19:57:02.339438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.339402 5733 scope.go:122] "RemoveContainer" containerID="06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995" Oct 30 19:57:02.339933 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:57:02.339901 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995\": container with ID starting with 06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995 not found: ID does not exist" containerID="06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995" Oct 30 19:57:02.340039 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.339960 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995"} err="failed to get container status \"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995\": rpc error: code = NotFound desc = could not find container \"06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995\": container with ID starting with 06908e32e3cf1f143bef638e006bb7f58a3748c725c346c8853f0652c0cc6995 not found: ID does not exist" Oct 30 19:57:02.359236 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.359151 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:57:02.370559 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.370501 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn"] Oct 30 19:57:02.772082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:02.770560 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.641290 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:03.636631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.264099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.263500 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.328898 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.328851 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" path="/var/lib/kubelet/pods/d3fcca0f-e02f-4881-89f6-16caf9ae4a5b/volumes" Oct 30 19:57:04.498076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.480234 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:57:04.498076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.480337 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:04.498076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.488326 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.689996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.689964 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:57:04.694961 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.694924 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:04.881110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:04.881049 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7f2a520-ccd657db9-w7skn" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.113:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:05.350080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:05.348627 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.983813 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:05.983160 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.531103 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:06.525928 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.367516 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:07.367480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.644586 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:08.644547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:09.709377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:09.709332 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.634692 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:10.634241 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.038556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:11.038464 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l"] Oct 30 19:57:11.043303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:11.043269 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d0e6762-68p2l"] Oct 30 19:57:11.636352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:11.636312 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:11.936279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:11.936192 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.052243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:12.052209 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d9f3fd-024d-4e24-97fa-cea7523aea17" path="/var/lib/kubelet/pods/e0d9f3fd-024d-4e24-97fa-cea7523aea17/volumes" Oct 30 19:57:12.177205 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:12.177173 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.293918 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.293880 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.546641 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.546530 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.637830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.637793 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 19:57:13.638584 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.638568 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerName="maas-api" Oct 30 19:57:13.638643 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.638585 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerName="maas-api" Oct 30 19:57:13.638698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.638689 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3fcca0f-e02f-4881-89f6-16caf9ae4a5b" containerName="maas-api" Oct 30 19:57:13.638733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.638701 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca917a2-706d-45df-ac03-51ec5ae9df0b" containerName="istio-proxy" Oct 30 19:57:13.643784 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.643754 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:13.651160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.651031 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b0d13a\"" Oct 30 19:57:13.669869 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.669840 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 19:57:13.701124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.701091 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b0d13a-dockercfg-bfmgl\"" Oct 30 19:57:13.739698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.739651 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttbt\" (UniqueName: \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:13.739884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.739779 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:13.773275 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.773239 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.841320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.841236 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttbt\" (UniqueName: \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:13.841667 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.841633 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:13.842624 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.842601 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:14.034120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:13.979820 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttbt\" (UniqueName: \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\") pod \"payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:14.160303 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:14.157521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.258074 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:14.258019 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:14.601006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:14.561522 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 19:57:14.837906 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:14.837874 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" event={"ID":"50a17c23-b0b0-4bd2-a9d2-82afacf2546d","Type":"ContainerStarted","Data":"8a96adcb46c35d6868efc93b5db223563fe7506c4734f2e5816cc74bc4dcbc3b"} Oct 30 19:57:14.944156 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:14.944118 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.847840 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:15.847807 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" event={"ID":"50a17c23-b0b0-4bd2-a9d2-82afacf2546d","Type":"ContainerStarted","Data":"188314e6a9045e9871b99971f0b2390a072bf291a8efb63d925c149b0d620fc1"} Oct 30 19:57:15.848233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:15.848014 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:15.889361 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:15.887385 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podStartSLOduration=2.887365616 podStartE2EDuration="2.887365616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:15.886324066 +0000 UTC m=+1595.954159647" watchObservedRunningTime="2026-10-30 19:57:15.887365616 +0000 UTC m=+1595.955201188" Oct 30 19:57:15.912637 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:15.908722 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.889089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:16.888322 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.807709 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:17.807665 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.041488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:18.041453 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.615081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:18.614077 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.849408 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:18.849368 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.301723 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:19.301683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.955548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:19.955515 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.162092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:20.161441 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.217662 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:21.217619 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.577395 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:21.577299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.752029 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:21.751989 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.634072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:22.634023 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.841627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:22.841587 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.866822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:23.866775 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.129816 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.129716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.186346 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.186313 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 19:57:24.191724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.191704 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.209670 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.209637 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 19:57:24.214845 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.214814 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae43d70e\"" Oct 30 19:57:24.311551 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.311518 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4rxz\" (UniqueName: \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.311890 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.311868 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.400481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.400350 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.417404 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.417355 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.417808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.417765 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4rxz\" (UniqueName: \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.423287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.423257 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.609180 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.601718 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 19:57:24.631074 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.631025 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 19:57:24.631323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.631167 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.631510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.631355 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4rxz\" (UniqueName: \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.671167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.640028 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae43d70e-dockercfg-mft54\"" Oct 30 19:57:24.731972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.731940 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk464\" (UniqueName: \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.732155 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.732049 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.808619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.808582 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:24.837201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.836158 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk464\" (UniqueName: \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.837201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.836235 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.837201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.837093 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.882272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.855300 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk464\" (UniqueName: \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:24.981374 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:24.981336 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:25.240545 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.240459 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 19:57:25.259335 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:57:25.259302 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3341ff20_d8d6_4a10_be1a_634df0c110cc.slice/crio-2269dbe7f1c3530a6d98f939d7d7735aa52960c6e9be8458d55161a19b28a5db WatchSource:0}: Error finding container 2269dbe7f1c3530a6d98f939d7d7735aa52960c6e9be8458d55161a19b28a5db: Status 404 returned error can't find the container with id 2269dbe7f1c3530a6d98f939d7d7735aa52960c6e9be8458d55161a19b28a5db Oct 30 19:57:25.397457 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:57:25.397186 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7473b6_ebba_4d4e_a552_a867d27fef42.slice/crio-fb8da1312654e0e7b1f7ac78ce675d5a8dbdff832ea7868b3af156c9d2d62ea6 WatchSource:0}: Error finding container fb8da1312654e0e7b1f7ac78ce675d5a8dbdff832ea7868b3af156c9d2d62ea6: Status 404 returned error can't find the container with id fb8da1312654e0e7b1f7ac78ce675d5a8dbdff832ea7868b3af156c9d2d62ea6 Oct 30 19:57:25.404913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.404882 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 19:57:25.645083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.640962 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.970352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.969975 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" event={"ID":"3341ff20-d8d6-4a10-be1a-634df0c110cc","Type":"ContainerStarted","Data":"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09"} Oct 30 19:57:25.970352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.970051 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" event={"ID":"3341ff20-d8d6-4a10-be1a-634df0c110cc","Type":"ContainerStarted","Data":"2269dbe7f1c3530a6d98f939d7d7735aa52960c6e9be8458d55161a19b28a5db"} Oct 30 19:57:25.970953 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.970911 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:25.973009 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.972985 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" event={"ID":"1a7473b6-ebba-4d4e-a552-a867d27fef42","Type":"ContainerStarted","Data":"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72"} Oct 30 19:57:25.973174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.973025 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" event={"ID":"1a7473b6-ebba-4d4e-a552-a867d27fef42","Type":"ContainerStarted","Data":"fb8da1312654e0e7b1f7ac78ce675d5a8dbdff832ea7868b3af156c9d2d62ea6"} Oct 30 19:57:25.973174 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:25.973051 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:26.021942 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.021887 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" podStartSLOduration=2.021868745 podStartE2EDuration="2.021868745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:26.014688135 +0000 UTC m=+1606.082523697" watchObservedRunningTime="2026-10-30 19:57:26.021868745 +0000 UTC m=+1606.089704314" Oct 30 19:57:26.047673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.047613 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" podStartSLOduration=2.047595695 podStartE2EDuration="2.047595695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:26.046648551 +0000 UTC m=+1606.114484119" watchObservedRunningTime="2026-10-30 19:57:26.047595695 +0000 UTC m=+1606.115431264" Oct 30 19:57:26.067034 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.067005 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd"] Oct 30 19:57:26.075875 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.075841 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0119ee5a-zwptd"] Oct 30 19:57:26.860320 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.860287 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 19:57:26.935803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:26.935770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.981078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:27.977326 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:28.055344 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:28.055305 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096" path="/var/lib/kubelet/pods/7a7d4b23-c74f-4f3c-8ad3-7eb9f6aeb096/volumes" Oct 30 19:57:28.906785 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:28.906745 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.069634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:29.069599 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.378437 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:57:29.378404 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:57:29.720585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:29.720491 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.558111 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:30.554021 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:31.318096 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:31.317986 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.122698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:32.122655 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.041051 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:33.041013 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.773309 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:33.773277 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.924481 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:33.924435 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.891746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:34.891708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:36.096821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:36.096781 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.024082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:37.016263 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:57:37.949080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:37.948646 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.015085 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:38.014283 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:57:38.443080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:38.442434 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.226774 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:39.226732 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.503009 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:40.502972 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:41.518081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:41.512217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.619595 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:42.619526 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.973325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:42.973238 5733 scope.go:122] "RemoveContainer" containerID="d4fedf545716c71fff73f7d73287b0eeb7542bdff217625fe1c525aff04f52de" Oct 30 19:57:42.984167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:42.984148 5733 scope.go:122] "RemoveContainer" containerID="46731d708d094bbb7759291f1b60400e0aee10244f9ed7f374797964c608ae40" Oct 30 19:57:43.030195 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:43.030164 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.298546 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:43.298470 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.900107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:43.889329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.226431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:44.226329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:44.425271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:44.425231 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:45.113206 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:45.113168 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.155663 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:46.155621 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.601885 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:47.601845 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.075837 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:48.075762 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.308239 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:48.308197 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.485100 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:48.485051 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.902082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:48.899662 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.236421 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.236338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.736219 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.736187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.808279 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.808247 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 19:57:49.814913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.814894 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:49.839889 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.839858 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3d0a78-dockercfg-wq8s9\"" Oct 30 19:57:49.843107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.843039 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 19:57:50.019135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.991400 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44mj\" (UniqueName: \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.019135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:49.991672 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.054863 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.054830 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3d0a78\"" Oct 30 19:57:50.109125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.100830 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44mj\" (UniqueName: \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.118962 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.109619 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.121092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.121035 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.144282 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.132314 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.166627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.166587 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44mj\" (UniqueName: \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\") pod \"payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.444081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.440385 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:50.873121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.873026 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 19:57:50.887099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:50.887027 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.583485 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.583455 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.651325 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.651289 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" event={"ID":"f4c14d49-8d28-428b-80a5-51a881aae5d1","Type":"ContainerStarted","Data":"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a"} Oct 30 19:57:51.651510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.651349 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" event={"ID":"f4c14d49-8d28-428b-80a5-51a881aae5d1","Type":"ContainerStarted","Data":"84bc632631a4c2019c56be2abce9766513e2ad3d20dbbe44bb7aa28b73223017"} Oct 30 19:57:51.651510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.651378 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:57:51.678188 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.678127 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podStartSLOduration=2.678112174 podStartE2EDuration="2.678112174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:51.674046682 +0000 UTC m=+1631.741882250" watchObservedRunningTime="2026-10-30 19:57:51.678112174 +0000 UTC m=+1631.745947789" Oct 30 19:57:51.908166 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:51.908134 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.906981 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:52.906948 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.159008 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:53.158915 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.889745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:53.889683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:54.986998 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:54.986958 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:55.209183 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:55.209132 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.040097 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:56.040021 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.977670 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:56.977631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.837076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:57.835459 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.798119 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:58.798084 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.379784 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:57:59.379749 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:57:59.588356 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:59.588313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.766285 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:57:59.766199 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.108242 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.108196 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.577952 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.577860 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 19:58:00.583568 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.583541 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.595472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.595439 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3d0a78-serving-cert\"" Oct 30 19:58:00.598844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.598815 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 19:58:00.642418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.642380 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.642605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.642549 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r669\" (UniqueName: \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.642605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.642594 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.642816 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.642798 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.754826 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.754778 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.755577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.755548 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r669\" (UniqueName: \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.755776 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.755758 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.756170 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.756128 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.756633 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:58:00.756614 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-serving-cert: secret "maas-api-e2e-shared-b-w0-3d0a78-serving-cert" not found Oct 30 19:58:00.756845 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:58:00.756832 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls podName:34580b12-bb45-4ca9-be6c-95b6421e8c0e nodeName:}" failed. No retries permitted until 2026-10-30 19:58:01.256793174 +0000 UTC m=+1641.324628740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls") pod "maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" (UID: "34580b12-bb45-4ca9-be6c-95b6421e8c0e") : secret "maas-api-e2e-shared-b-w0-3d0a78-serving-cert" not found Oct 30 19:58:00.758006 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.757984 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.776152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.773508 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.776152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.775401 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.795026 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.792248 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r669\" (UniqueName: \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:00.916363 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:00.916323 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.120646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.120617 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.263982 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.263884 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:01.270528 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.270504 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:01.508287 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.508256 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:01.973311 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.973265 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 19:58:01.974130 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:01.974095 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34580b12_bb45_4ca9_be6c_95b6421e8c0e.slice/crio-bfbd8726a1ff7919f330a1e106e438871077e33ddc5bef00ffc1f2ef255a767e WatchSource:0}: Error finding container bfbd8726a1ff7919f330a1e106e438871077e33ddc5bef00ffc1f2ef255a767e: Status 404 returned error can't find the container with id bfbd8726a1ff7919f330a1e106e438871077e33ddc5bef00ffc1f2ef255a767e Oct 30 19:58:01.976042 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:01.976018 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:58:02.091016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.090978 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:02.662965 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.662938 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 19:58:02.797489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.797394 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerStarted","Data":"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75"} Oct 30 19:58:02.797489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.797458 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerStarted","Data":"bfbd8726a1ff7919f330a1e106e438871077e33ddc5bef00ffc1f2ef255a767e"} Oct 30 19:58:02.797489 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.797487 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:02.824958 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:02.824900 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podStartSLOduration=2.352262157 podStartE2EDuration="2.824883286s" totalImagesPullingTime="472.621129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.824477028 +0000 UTC m=+1642.892312597" watchObservedRunningTime="2026-10-30 19:58:02.824883286 +0000 UTC m=+1642.892718856" Oct 30 19:58:03.288697 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:03.288657 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.300759 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:04.300724 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.357750 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:05.357706 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:06.550431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:06.550393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.485225 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:07.485168 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:08.314609 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:08.314536 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:08.813107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:08.813076 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 19:58:09.266072 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:09.266024 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:10.004794 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:10.004761 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:10.903908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:10.903867 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.789733 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:11.789694 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.147414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:12.147378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.931349 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:12.931314 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.207417 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:13.207318 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.037760 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:14.037688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:15.151112 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:15.147359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:16.462698 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:16.462662 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:16.838908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:16.838796 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.204467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:17.204431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.523277 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:17.523166 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.724773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:17.724689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:18.703651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.703611 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:18.760747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.760714 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 19:58:18.766414 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.766389 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:18.779934 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.779913 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-274922\"" Oct 30 19:58:18.788341 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.788317 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 19:58:18.857713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.857675 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:18.857910 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.857813 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqls\" (UniqueName: \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:18.969367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.969262 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:18.970676 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.970650 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqls\" (UniqueName: \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:18.977696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:18.970515 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:19.010014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.009968 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqls\" (UniqueName: \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\") pod \"payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:19.047699 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.047603 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.089220 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.089191 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:19.359612 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.358307 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 19:58:19.365778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.365747 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.379782 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.379753 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b0d13a-openshift-default-dockercfg-b2mpb\"" Oct 30 19:58:19.403908 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.403873 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 19:58:19.516924 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.516833 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.516924 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.516881 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517010 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517186 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517161 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517243 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517302 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517285 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517467 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517444 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517607 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zs7h\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.517675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.517645 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.539109 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.539074 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 19:58:19.563934 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:19.563906 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66363c5c_5f40_47f5_bb62_4b00184dba74.slice/crio-2b3c6a65744dc01af3021a6563d2173242eae1b6cf1ea3b46d83aaa6da802b9c WatchSource:0}: Error finding container 2b3c6a65744dc01af3021a6563d2173242eae1b6cf1ea3b46d83aaa6da802b9c: Status 404 returned error can't find the container with id 2b3c6a65744dc01af3021a6563d2173242eae1b6cf1ea3b46d83aaa6da802b9c Oct 30 19:58:19.619299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619271 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619313 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619434 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619390 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619565 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619483 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619628 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619608 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619686 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619657 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619740 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619707 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619746 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619879 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619857 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.619999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.619975 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.620149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.620022 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zs7h\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.620149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.620097 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.620488 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.620463 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.620710 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.620688 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.622300 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.622276 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.623391 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.623372 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.638872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.638844 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.672197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.672167 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zs7h\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:19.687154 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:19.687129 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:20.022846 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.022815 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 19:58:20.025008 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:20.024966 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff87fcd9_988d_4f08_aaf3_86d2ff50bab6.slice/crio-504b6f9d4e30c2793eb81b1df20c1639dc0bb4dde41328023499be9063e29998 WatchSource:0}: Error finding container 504b6f9d4e30c2793eb81b1df20c1639dc0bb4dde41328023499be9063e29998: Status 404 returned error can't find the container with id 504b6f9d4e30c2793eb81b1df20c1639dc0bb4dde41328023499be9063e29998 Oct 30 19:58:20.030021 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.029988 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:20.030146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.030083 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:20.030146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.030130 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:20.172577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.172531 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" event={"ID":"66363c5c-5f40-47f5-bb62-4b00184dba74","Type":"ContainerStarted","Data":"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2"} Oct 30 19:58:20.172753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.172615 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" event={"ID":"66363c5c-5f40-47f5-bb62-4b00184dba74","Type":"ContainerStarted","Data":"2b3c6a65744dc01af3021a6563d2173242eae1b6cf1ea3b46d83aaa6da802b9c"} Oct 30 19:58:20.172753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.172669 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:20.174585 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.174550 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" event={"ID":"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6","Type":"ContainerStarted","Data":"e74afd51df82be2060435936a8865f5c635ccc3f2a65d4501a943bef6e90e7e0"} Oct 30 19:58:20.174703 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.174599 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" event={"ID":"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6","Type":"ContainerStarted","Data":"504b6f9d4e30c2793eb81b1df20c1639dc0bb4dde41328023499be9063e29998"} Oct 30 19:58:20.197029 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.196975 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podStartSLOduration=2.196959067 podStartE2EDuration="2.196959067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:20.196335615 +0000 UTC m=+1660.264171182" watchObservedRunningTime="2026-10-30 19:58:20.196959067 +0000 UTC m=+1660.264794638" Oct 30 19:58:20.231082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.230406 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podStartSLOduration=1.230384539 podStartE2EDuration="1.230384539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:20.229034218 +0000 UTC m=+1660.296869787" watchObservedRunningTime="2026-10-30 19:58:20.230384539 +0000 UTC m=+1660.298220109" Oct 30 19:58:20.423248 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.423205 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.647847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.647811 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.687882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:20.687804 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:21.102280 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.102239 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 19:58:21.172927 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.172857 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:21.209746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.209710 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.264082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.260621 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bdd3b5-openshift-default-dockercfg-mbzvc\"" Oct 30 19:58:21.416089 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.409109 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 19:58:21.540080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.512907 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:21.707274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707172 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707274 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707242 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707497 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707482 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707518 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707550 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707675 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707623 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707731 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707715 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707792 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707744 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.707847 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.707822 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdkq\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.714790 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.714744 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:21.715074 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.715030 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:21.809286 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809255 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdkq\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809419 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809464 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809737 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809850 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809784 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809850 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809817 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.809963 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.809918 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.810075 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.810035 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.821172 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.821138 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.821497 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.810221 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.821689 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.810289 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.821796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.810189 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.830433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.830396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.830871 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.830846 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.852263 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.852221 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.854318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.854279 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdkq\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.856172 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.856144 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.856531 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.856511 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\") pod \"e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.884159 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.884131 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:21.956693 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:21.956608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.289688 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.289314 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.669891 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.669863 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 19:58:22.671580 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.671547 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:22.671671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.671615 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:22.671671 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.671645 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:58:22.689308 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.689285 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:22.689410 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.689330 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:22.967135 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:22.965447 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.496551 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.496522 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.588448 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.586038 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerStarted","Data":"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e"} Oct 30 19:58:23.588448 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.586154 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerStarted","Data":"68e0ac5cc46595fce4ed64b9aed7b4d874f27fbbb39106832dbcd217cb2a258b"} Oct 30 19:58:23.657642 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.657516 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podStartSLOduration=2.657492527 podStartE2EDuration="2.657492527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:23.642586711 +0000 UTC m=+1663.710422280" watchObservedRunningTime="2026-10-30 19:58:23.657492527 +0000 UTC m=+1663.725328096" Oct 30 19:58:23.687629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.687598 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:23.687874 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.687851 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:23.781197 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.776946 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:23.888081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.887352 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:23.942082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:23.937779 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:24.594117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:24.594084 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:24.594926 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:24.594906 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:58:24.687842 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:24.687806 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:24.687990 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:24.687858 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:25.051173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:25.051040 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:25.696937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:25.696007 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:25.696937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:25.696083 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:26.265083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:26.264578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:26.554121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:26.553883 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:26.687713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:26.687682 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 30 19:58:26.687879 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:26.687733 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 30 19:58:26.835121 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:26.835014 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:27.693474 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:27.693444 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:27.694913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:27.694889 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:27.695940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:27.695917 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 19:58:28.040646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.040551 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:28.118640 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.118603 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 19:58:28.123603 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.123579 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.140777 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.140748 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 19:58:28.141211 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.141187 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-bdd3b5\"" Oct 30 19:58:28.143659 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.143598 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-bdd3b5-dockercfg-mn8fr\"" Oct 30 19:58:28.226238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.223988 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hxx\" (UniqueName: \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.226238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.224197 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.428158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.407619 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:28.461902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.457241 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hxx\" (UniqueName: \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.461902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.457453 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.507214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.507174 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hxx\" (UniqueName: \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.507707 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.507677 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.518105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.512545 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 19:58:28.546110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.543077 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.546110 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.543930 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 19:58:28.664822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.662870 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.664822 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.663121 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgsk\" (UniqueName: \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.686646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.686541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:28.749359 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.749321 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:28.765305 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.765249 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.765509 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.765489 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgsk\" (UniqueName: \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.766225 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.766194 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.794107 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.791316 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgsk\" (UniqueName: \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\") pod \"payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:28.900673 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:28.900642 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:29.013747 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.013629 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.096957 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.096922 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 19:58:29.098262 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:29.098233 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6844dca4_0d51_4f57_911f_7238e5deebc6.slice/crio-ccf6b8be945c54348c90b124b4f2bd43fcb301c7b53db9ea07bc19d2baae1e5f WatchSource:0}: Error finding container ccf6b8be945c54348c90b124b4f2bd43fcb301c7b53db9ea07bc19d2baae1e5f: Status 404 returned error can't find the container with id ccf6b8be945c54348c90b124b4f2bd43fcb301c7b53db9ea07bc19d2baae1e5f Oct 30 19:58:29.224560 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.224531 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 19:58:29.224991 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:29.224949 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d4bbb7_e918_421a_859d_0e1c072ee7fd.slice/crio-6aa7e32af4af7505c80b3992a1196f17a56c947ba5c9f45da828565490cdaa0a WatchSource:0}: Error finding container 6aa7e32af4af7505c80b3992a1196f17a56c947ba5c9f45da828565490cdaa0a: Status 404 returned error can't find the container with id 6aa7e32af4af7505c80b3992a1196f17a56c947ba5c9f45da828565490cdaa0a Oct 30 19:58:29.410128 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:58:29.405435 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:58:29.413517 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.413439 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.716084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.715991 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" event={"ID":"6844dca4-0d51-4f57-911f-7238e5deebc6","Type":"ContainerStarted","Data":"7ae9ab0841d0b01299c4283fb3ef23717984d92af0e04ce55ef204f3f8a0ebc7"} Oct 30 19:58:29.716292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.716115 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:29.716292 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.716149 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" event={"ID":"6844dca4-0d51-4f57-911f-7238e5deebc6","Type":"ContainerStarted","Data":"ccf6b8be945c54348c90b124b4f2bd43fcb301c7b53db9ea07bc19d2baae1e5f"} Oct 30 19:58:29.718384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.718348 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" event={"ID":"e4d4bbb7-e918-421a-859d-0e1c072ee7fd","Type":"ContainerStarted","Data":"39ffabc117d6fb2672dd3348cd1e391e49e5d68b7c55cd3fca32be370e2d0754"} Oct 30 19:58:29.718542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.718407 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" event={"ID":"e4d4bbb7-e918-421a-859d-0e1c072ee7fd","Type":"ContainerStarted","Data":"6aa7e32af4af7505c80b3992a1196f17a56c947ba5c9f45da828565490cdaa0a"} Oct 30 19:58:29.718542 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.718441 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:29.752084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.751696 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" podStartSLOduration=1.751675997 podStartE2EDuration="1.751675997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.749116 +0000 UTC m=+1669.816951569" watchObservedRunningTime="2026-10-30 19:58:29.751675997 +0000 UTC m=+1669.819511565" Oct 30 19:58:29.777946 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:29.777887 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" podStartSLOduration=1.777871616 podStartE2EDuration="1.777871616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.77434564 +0000 UTC m=+1669.842181208" watchObservedRunningTime="2026-10-30 19:58:29.777871616 +0000 UTC m=+1669.845707184" Oct 30 19:58:30.639124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:30.635440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:31.605831 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:31.605800 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 19:58:31.629007 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:31.628963 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:31.998746 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:31.998656 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:32.085386 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:32.085344 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv"] Oct 30 19:58:32.094020 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:32.093987 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9b928fd-jgzgv"] Oct 30 19:58:32.685078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:32.683136 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.473042 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:33.472951 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:34.052871 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:34.052829 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8608f56-0718-4176-8ff2-52cca3bbabb7" path="/var/lib/kubelet/pods/e8608f56-0718-4176-8ff2-52cca3bbabb7/volumes" Oct 30 19:58:34.379843 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:34.379805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:35.305167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:35.305130 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:35.483084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:35.482749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:36.570708 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:36.570671 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:37.868700 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:37.868647 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.911921 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:38.911883 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.105424 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:39.105392 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.288771 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:39.288685 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.218040 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:40.217309 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.733184 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:40.733156 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 19:58:40.733367 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:40.733228 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 19:58:41.005134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:41.005025 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.033075 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.033023 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.708648 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.708619 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 19:58:42.715099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.715073 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.725176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.725146 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bdd3b5-serving-cert\"" Oct 30 19:58:42.731531 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.730874 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 19:58:42.819808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.819770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.825431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.825405 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.825639 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.825616 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.825766 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.825747 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwl78\" (UniqueName: \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.825809 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.825794 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.936116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.929743 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.936116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.929905 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwl78\" (UniqueName: \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.936116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.929965 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.936116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.933414 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.945684 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.945652 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.963123 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.963034 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.968048 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.967978 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwl78\" (UniqueName: \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.968655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.968623 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\") pod \"maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:42.995005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:42.994973 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:43.028137 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.028043 5733 scope.go:122] "RemoveContainer" containerID="58586b4ad392a0d6c2d88bd07fad9ca0d2aa701712af18ece4dcade426649d11" Oct 30 19:58:43.042016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.041985 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:43.043142 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.043124 5733 scope.go:122] "RemoveContainer" containerID="5aa2458f9fc1c8b41ce638f6ec4488278e0f3c3811a8db1a25a6e8f3f03dd9a4" Oct 30 19:58:43.058752 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.058727 5733 scope.go:122] "RemoveContainer" containerID="b8dc644b239af082bd0cf986ee45b20e8f0d8119954d345e1fc47d1cc2b24813" Oct 30 19:58:43.092372 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.092348 5733 scope.go:122] "RemoveContainer" containerID="53b9ec62a39c0b9fc9e3eadb360f823dc42cf1958a6f2e21624b87bc50e8f1d8" Oct 30 19:58:43.354910 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.354879 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 19:58:43.989753 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.989713 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" event={"ID":"4c46ab70-aa39-46da-ae9d-28abcbaf69ba","Type":"ContainerStarted","Data":"a4f2e35376a2df443eaa9c5042e714cf1106a70c8dac90e172afc9dc51262023"} Oct 30 19:58:43.989935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.989781 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" event={"ID":"4c46ab70-aa39-46da-ae9d-28abcbaf69ba","Type":"ContainerStarted","Data":"0a5806388198130d9ac59b3e2312d00d8f2981652901701838312b9e55a257db"} Oct 30 19:58:43.989935 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:43.989907 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:44.025724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:44.025688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:44.028744 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:44.028691 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" podStartSLOduration=1.561059789 podStartE2EDuration="2.028675034s" totalImagesPullingTime="467.615245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:44.023797129 +0000 UTC m=+1684.091632700" watchObservedRunningTime="2026-10-30 19:58:44.028675034 +0000 UTC m=+1684.096510605" Oct 30 19:58:45.049638 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:45.044881 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.088020 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:46.087981 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.062608 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:47.062568 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.070431 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:47.070394 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6"] Oct 30 19:58:47.076901 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:47.076869 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f3744e-q49h6"] Oct 30 19:58:48.013410 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:48.013374 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.055803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:48.055768 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701faa90-8cba-4b4e-a189-ab1db587a0c0" path="/var/lib/kubelet/pods/701faa90-8cba-4b4e-a189-ab1db587a0c0/volumes" Oct 30 19:58:48.944527 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:48.944479 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.727881 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:49.727844 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:50.041913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:50.041836 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 19:58:50.338157 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:50.338010 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:50.859969 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:50.859936 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:51.161153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:51.159393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:51.639345 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:51.639307 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:51.980148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:51.972686 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.234618 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:53.234567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.214082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:54.210322 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.806318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:54.806283 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:54.999655 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:54.999622 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.332145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:55.332111 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.400544 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.400509 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 19:58:56.402627 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.402601 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.406920 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.406899 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.420712 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.420691 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-98a0d8a9-dockercfg-6pssf\"" Oct 30 19:58:56.420855 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.420758 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-98a0d8a9\"" Oct 30 19:58:56.426816 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.426790 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 19:58:56.474720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.474672 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.475402 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.474925 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk2pz\" (UniqueName: \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.611363 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.603652 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk2pz\" (UniqueName: \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.611363 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.603856 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.657149 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.652846 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.767130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.766938 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.822081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.789734 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 19:58:56.847143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.841822 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk2pz\" (UniqueName: \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\") pod \"payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:56.868138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.861649 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:56.977981 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:56.977700 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 19:58:57.035078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.034376 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:57.054867 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.054825 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.055163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.055132 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrxt\" (UniqueName: \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.207644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.205679 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.207644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.205739 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrxt\" (UniqueName: \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.207644 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.207563 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.237134 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.233991 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrxt\" (UniqueName: \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\") pod \"payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.326724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.326467 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:57.492299 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.489512 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:57.587420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.587318 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 19:58:57.670497 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.670468 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 19:58:57.670686 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:58:57.670665 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e05429_d1e2_424a_8059_3c949fb1a083.slice/crio-06606d3d404abdf8fc108dc6ceb89f0678699c1a814e04030756f731b6f3a76d WatchSource:0}: Error finding container 06606d3d404abdf8fc108dc6ceb89f0678699c1a814e04030756f731b6f3a76d: Status 404 returned error can't find the container with id 06606d3d404abdf8fc108dc6ceb89f0678699c1a814e04030756f731b6f3a76d Oct 30 19:58:57.906004 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:57.904418 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.286696 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.286662 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" event={"ID":"07e05429-d1e2-424a-8059-3c949fb1a083","Type":"ContainerStarted","Data":"552768eebf3ead751e4620e35da8985fb78caeca73104fbc37b257b2700baebc"} Oct 30 19:58:58.286894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.286719 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" event={"ID":"07e05429-d1e2-424a-8059-3c949fb1a083","Type":"ContainerStarted","Data":"06606d3d404abdf8fc108dc6ceb89f0678699c1a814e04030756f731b6f3a76d"} Oct 30 19:58:58.286894 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.286749 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:58:58.288496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.288464 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" event={"ID":"274c9fcd-f20f-4e00-837f-17e92dc7820d","Type":"ContainerStarted","Data":"599e0aa4849ad6a7ad90c3d5ffdc64d3c558fe7979ad8555191b50deed541cbf"} Oct 30 19:58:58.288631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.288524 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:58:58.288631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.288551 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" event={"ID":"274c9fcd-f20f-4e00-837f-17e92dc7820d","Type":"ContainerStarted","Data":"0ef8fea051521e4a8491fa36e4cb837ff4f4878f5c4d13a40353ddb751bd6897"} Oct 30 19:58:58.317647 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.317590 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" podStartSLOduration=2.3175733640000002 podStartE2EDuration="2.317573364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.315860121 +0000 UTC m=+1698.383695691" watchObservedRunningTime="2026-10-30 19:58:58.317573364 +0000 UTC m=+1698.385408932" Oct 30 19:58:58.363401 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.363336 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" podStartSLOduration=2.363317233 podStartE2EDuration="2.363317233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.359413051 +0000 UTC m=+1698.427248620" watchObservedRunningTime="2026-10-30 19:58:58.363317233 +0000 UTC m=+1698.431152801" Oct 30 19:58:58.411296 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.411258 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.611748 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.611708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.933144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:58.923858 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.094932 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:59.088086 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.384477 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:58:59.384440 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:58:59.870271 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:58:59.870241 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.752079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:00.744979 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:01.044720 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:01.044626 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z"] Oct 30 19:59:01.052152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:01.052118 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b02b572-4sc6z"] Oct 30 19:59:01.771080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:01.767732 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.054207 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:02.054105 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58574de6-b277-4383-bc54-a9292a4191a9" path="/var/lib/kubelet/pods/58574de6-b277-4383-bc54-a9292a4191a9/volumes" Oct 30 19:59:02.253338 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:02.247564 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.903388 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:02.903348 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:03.086602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:03.086553 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:03.352141 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:03.337135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:03.749477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:03.749391 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.415226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.415160 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 19:59:04.421930 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.421904 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.472555 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.471461 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.473685 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.473629 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 19:59:04.572593 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572550 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.572755 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572620 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.572840 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572742 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.572840 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572818 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.572952 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572907 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.573012 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.572981 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.573201 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.573168 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.573289 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.573247 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.573360 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.573337 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6b7m\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.667586 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.667494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.675214 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675181 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675379 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675260 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675379 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675358 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6b7m\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675510 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675494 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675577 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675561 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675598 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675635 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675624 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675740 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675721 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675788 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675821 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675806 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675929 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675886 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.675986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.675934 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.676301 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.676276 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.676429 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.676408 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.677938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.677914 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.677938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.677927 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.698203 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.698177 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6b7m\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.699713 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.699690 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\") pod \"e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.763937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.763900 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:04.904294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:04.904230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:05.135124 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:05.127079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:05.421226 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:05.421194 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 19:59:05.422692 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:05.422652 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52995638_b72e_460b_8306_f442c213ff19.slice/crio-a51ca5669309ac508fbdd2c06129d1c0c12da144a05731785944f8a3eef40cff WatchSource:0}: Error finding container a51ca5669309ac508fbdd2c06129d1c0c12da144a05731785944f8a3eef40cff: Status 404 returned error can't find the container with id a51ca5669309ac508fbdd2c06129d1c0c12da144a05731785944f8a3eef40cff Oct 30 19:59:05.426200 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:05.426164 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:05.426294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:05.426237 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:05.426294 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:05.426267 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:06.257778 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.257744 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.408857 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.408819 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" event={"ID":"52995638-b72e-460b-8306-f442c213ff19","Type":"ContainerStarted","Data":"df3346d191c5ea28010f0a63c829ddc9995698d9f92f0081cf32b9f2acd89225"} Oct 30 19:59:06.409023 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.408881 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" event={"ID":"52995638-b72e-460b-8306-f442c213ff19","Type":"ContainerStarted","Data":"a51ca5669309ac508fbdd2c06129d1c0c12da144a05731785944f8a3eef40cff"} Oct 30 19:59:06.768418 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.768381 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:06.769032 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.768577 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:06.769032 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.768634 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:06.903736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:06.903691 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.566575 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:07.566537 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.764473 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:07.764436 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:07.764745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:07.764511 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:07.857494 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:07.857462 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:08.239140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:08.203730 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:08.615016 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:08.614980 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:08.764526 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:08.764486 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:08.764705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:08.764555 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:08.864352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:08.864304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.049995 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.049899 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.303283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.303204 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 19:59:09.303970 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.303945 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 19:59:09.399117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.399019 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podStartSLOduration=5.399004617 podStartE2EDuration="5.399004617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:06.451374145 +0000 UTC m=+1706.519209717" watchObservedRunningTime="2026-10-30 19:59:09.399004617 +0000 UTC m=+1709.466840187" Oct 30 19:59:09.773295 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.773257 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:09.773588 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:09.773556 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:10.122668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.121650 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.398567 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.398487 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.642145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.642107 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.755382 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.755285 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 19:59:10.761509 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.761481 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:10.769106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.768704 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:10.769106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.768755 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:10.829088 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.808977 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 19:59:10.879084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.873824 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c4c072\"" Oct 30 19:59:10.879084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.873997 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c4c072-dockercfg-cbmt4\"" Oct 30 19:59:10.925770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.925726 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.001146 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:10.994898 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.084143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.083049 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfl4l\" (UniqueName: \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.202830 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.202776 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.208373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.203141 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfl4l\" (UniqueName: \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.208373 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.207833 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.268086 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.267489 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfl4l\" (UniqueName: \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\") pod \"payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.441117 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.440362 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:11.808083 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.783180 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Oct 30 19:59:11.808420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:11.808383 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Oct 30 19:59:12.342280 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.325352 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 19:59:12.384480 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:12.377912 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ad9fc8_b478_4513_bf7c_76983ae03126.slice/crio-86da29b7ee9bf3b9794f3f3f195d50064f2bec971c197d95b1bb241f5ca3ffa2 WatchSource:0}: Error finding container 86da29b7ee9bf3b9794f3f3f195d50064f2bec971c197d95b1bb241f5ca3ffa2: Status 404 returned error can't find the container with id 86da29b7ee9bf3b9794f3f3f195d50064f2bec971c197d95b1bb241f5ca3ffa2 Oct 30 19:59:12.895017 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.894973 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" event={"ID":"16ad9fc8-b478-4513-bf7c-76983ae03126","Type":"ContainerStarted","Data":"b43186f22064504a0ad2f97f66f4e4e915254181c3b0230d074dc779eccf12ab"} Oct 30 19:59:12.895433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.895410 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:12.895531 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.895454 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" event={"ID":"16ad9fc8-b478-4513-bf7c-76983ae03126","Type":"ContainerStarted","Data":"86da29b7ee9bf3b9794f3f3f195d50064f2bec971c197d95b1bb241f5ca3ffa2"} Oct 30 19:59:12.947014 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.946944 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" podStartSLOduration=2.946925219 podStartE2EDuration="2.946925219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:12.934393477 +0000 UTC m=+1713.002229061" watchObservedRunningTime="2026-10-30 19:59:12.946925219 +0000 UTC m=+1713.014760789" Oct 30 19:59:12.969153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.954780 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:12.969153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.961631 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:12.969153 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:12.967388 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 19:59:13.203163 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:13.203036 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:13.686986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:13.686946 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.786384 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.786335 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.859150 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.859109 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 19:59:14.870811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.870778 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 19:59:14.870994 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.870907 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:14.886480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.883945 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-dd1b04\"" Oct 30 19:59:14.886480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.884681 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 19:59:14.886480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.884722 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 19:59:14.886480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.884826 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:14.886480 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.885455 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-dd1b04-dockercfg-jvdqc\"" Oct 30 19:59:14.975266 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:14.975225 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:15.065130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.064404 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qs6k\" (UniqueName: \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.065130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.064473 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbrn\" (UniqueName: \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.065130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.064566 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.065130 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.064621 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.323536 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.283975 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbrn\" (UniqueName: \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.359243 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.352890 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.406171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.356600 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:15.406171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.405456 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.406171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.405559 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.406171 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.406098 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qs6k\" (UniqueName: \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.410447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.407532 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.423077 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.416030 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbrn\" (UniqueName: \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\") pod \"payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.466078 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.464770 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qs6k\" (UniqueName: \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\") pod \"payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.534098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.501262 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:15.534098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.509477 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:15.984479 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:15.984430 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3c6bd6_14b6_499f_abf6_99ff8999eee8.slice/crio-cacec2e14ed65fcbfe88238daf7417795d73645808375cdbf12cc32ad3086eaf WatchSource:0}: Error finding container cacec2e14ed65fcbfe88238daf7417795d73645808375cdbf12cc32ad3086eaf: Status 404 returned error can't find the container with id cacec2e14ed65fcbfe88238daf7417795d73645808375cdbf12cc32ad3086eaf Oct 30 19:59:15.984899 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.984859 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 19:59:15.988406 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:15.988376 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1dd9f_fc5b_406b_9a08_769097ed3528.slice/crio-995dd955190deaa7fd259a36ce24f492c58e5deba3bcb1653623e76b9f9742ba WatchSource:0}: Error finding container 995dd955190deaa7fd259a36ce24f492c58e5deba3bcb1653623e76b9f9742ba: Status 404 returned error can't find the container with id 995dd955190deaa7fd259a36ce24f492c58e5deba3bcb1653623e76b9f9742ba Oct 30 19:59:15.990614 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:15.990589 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 19:59:16.257024 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.256937 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.974999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.974939 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" event={"ID":"0fb1dd9f-fc5b-406b-9a08-769097ed3528","Type":"ContainerStarted","Data":"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510"} Oct 30 19:59:16.975456 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.975027 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" event={"ID":"0fb1dd9f-fc5b-406b-9a08-769097ed3528","Type":"ContainerStarted","Data":"995dd955190deaa7fd259a36ce24f492c58e5deba3bcb1653623e76b9f9742ba"} Oct 30 19:59:16.975456 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.975100 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:16.976967 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.976944 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" event={"ID":"ab3c6bd6-14b6-499f-abf6-99ff8999eee8","Type":"ContainerStarted","Data":"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3"} Oct 30 19:59:16.977131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.976980 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" event={"ID":"ab3c6bd6-14b6-499f-abf6-99ff8999eee8","Type":"ContainerStarted","Data":"cacec2e14ed65fcbfe88238daf7417795d73645808375cdbf12cc32ad3086eaf"} Oct 30 19:59:16.977131 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:16.977052 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:17.006114 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:17.006042 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" podStartSLOduration=3.006025039 podStartE2EDuration="3.006025039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:16.998021949 +0000 UTC m=+1717.065857521" watchObservedRunningTime="2026-10-30 19:59:17.006025039 +0000 UTC m=+1717.073860607" Oct 30 19:59:17.089125 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:17.089022 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" podStartSLOduration=3.089003081 podStartE2EDuration="3.089003081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:17.077886725 +0000 UTC m=+1717.145722295" watchObservedRunningTime="2026-10-30 19:59:17.089003081 +0000 UTC m=+1717.156838650" Oct 30 19:59:17.346687 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:17.346648 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:17.739144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:17.735932 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.086833 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.086750 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd"] Oct 30 19:59:18.094862 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.094835 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx"] Oct 30 19:59:18.102152 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.102129 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de07ab40-h5hqd"] Oct 30 19:59:18.108160 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.108135 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bfd631a-gh8rx"] Oct 30 19:59:18.388362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.388327 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 19:59:18.394469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.394435 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.409653 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.409612 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-98a0d8a9-serving-cert\"" Oct 30 19:59:18.415616 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.415588 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.418517 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.418489 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.419098 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.419047 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49gqw\" (UniqueName: \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.419346 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.419324 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.428419 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.428395 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 19:59:18.485472 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.485437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.524884 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.524844 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.525469 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.525443 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49gqw\" (UniqueName: \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.525718 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.525683 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.525849 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.525788 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.526077 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:59:18.526042 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-serving-cert: secret "maas-api-e2e-mt-98a0d8a9-serving-cert" not found Oct 30 19:59:18.526235 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:59:18.526223 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls podName:b9bc9338-e466-4f64-a5ce-5cc163ac9ca5 nodeName:}" failed. No retries permitted until 2026-10-30 19:59:19.026201761 +0000 UTC m=+1719.094037325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls") pod "maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" (UID: "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5") : secret "maas-api-e2e-mt-98a0d8a9-serving-cert" not found Oct 30 19:59:18.527118 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.527097 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.580811 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.577149 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.603095 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.601003 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49gqw\" (UniqueName: \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:18.670122 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:18.669991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.030222 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.030138 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.054868 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.054384 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:19.062120 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.062045 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") pod \"maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:19.327913 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.327813 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:19.699583 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:19.699549 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bc9338_e466_4f64_a5ce_5cc163ac9ca5.slice/crio-6b0ee1e5a8ddb771e1729d2f154a06367cc28e388e5e22b73b9309707161837e WatchSource:0}: Error finding container 6b0ee1e5a8ddb771e1729d2f154a06367cc28e388e5e22b73b9309707161837e: Status 404 returned error can't find the container with id 6b0ee1e5a8ddb771e1729d2f154a06367cc28e388e5e22b73b9309707161837e Oct 30 19:59:19.705751 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.705723 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 19:59:19.833548 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:19.833469 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.021634 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:20.021534 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" event={"ID":"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5","Type":"ContainerStarted","Data":"6b0ee1e5a8ddb771e1729d2f154a06367cc28e388e5e22b73b9309707161837e"} Oct 30 19:59:20.062444 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:20.062414 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57288644-cfa7-44a9-bf39-43ecd96d7939" path="/var/lib/kubelet/pods/57288644-cfa7-44a9-bf39-43ecd96d7939/volumes" Oct 30 19:59:20.063233 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:20.063212 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6947777b-f683-42d2-a03f-3b0c838bef21" path="/var/lib/kubelet/pods/6947777b-f683-42d2-a03f-3b0c838bef21/volumes" Oct 30 19:59:20.778796 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:20.778756 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.033515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:21.033385 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" event={"ID":"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5","Type":"ContainerStarted","Data":"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a"} Oct 30 19:59:21.033515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:21.033460 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:21.057462 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:21.057400 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" podStartSLOduration=2.602833799 podStartE2EDuration="3.057382101s" totalImagesPullingTime="454.548302ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:21.055799046 +0000 UTC m=+1721.123634614" watchObservedRunningTime="2026-10-30 19:59:21.057382101 +0000 UTC m=+1721.125217669" Oct 30 19:59:21.688728 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:21.688694 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.272353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:22.272323 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.560872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:22.559514 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.870592 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:22.870552 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 19:59:22.947158 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:22.947119 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 19:59:22.947353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:22.947197 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056349 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056405 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056521 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056568 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb58c\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056723 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056827 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.063005 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.056940 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.158969 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.158871 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159001 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159032 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159154 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159201 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb58c\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159381 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159346 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159492 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159442 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.159976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.159942 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.162258 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.162238 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.162353 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.162255 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.162463 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.162443 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.162502 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.162481 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.175433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.175409 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb58c\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.176161 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.176138 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-g9dvq\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.302844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.302806 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:23.794116 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.794027 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.907773 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:23.907747 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 19:59:24.274515 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:24.274212 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0466fa1a_6c5c_4a43_a71d_42b4d000b07c.slice/crio-0fe2404807e4ef108de434a8e4ba2eb6417c1021a43a1f54aa55d6745fa26d22 WatchSource:0}: Error finding container 0fe2404807e4ef108de434a8e4ba2eb6417c1021a43a1f54aa55d6745fa26d22: Status 404 returned error can't find the container with id 0fe2404807e4ef108de434a8e4ba2eb6417c1021a43a1f54aa55d6745fa26d22 Oct 30 19:59:24.274515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.274456 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 19:59:24.277702 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.277667 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:24.277815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.277746 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:24.343386 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.343352 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr"] Oct 30 19:59:24.709802 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.709772 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr"] Oct 30 19:59:24.710000 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.709970 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.715323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.714910 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:24.715323 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.714995 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:59:24.793145 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.793108 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsfc\" (UniqueName: \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.793523 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.793499 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.793658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.793641 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.894915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.894882 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.895105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.894992 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.895105 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.895089 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsfc\" (UniqueName: \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.895924 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.895894 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.964514 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.964428 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.964803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.964783 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsfc\" (UniqueName: \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\") pod \"maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:24.970583 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:24.970544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:25.102050 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.102010 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" event={"ID":"0466fa1a-6c5c-4a43-a71d-42b4d000b07c","Type":"ContainerStarted","Data":"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489"} Oct 30 19:59:25.102293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.102113 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" event={"ID":"0466fa1a-6c5c-4a43-a71d-42b4d000b07c","Type":"ContainerStarted","Data":"0fe2404807e4ef108de434a8e4ba2eb6417c1021a43a1f54aa55d6745fa26d22"} Oct 30 19:59:25.102293 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.102146 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:25.229355 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.229267 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:25.247196 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.219595 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g9dvq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:8080/ready\": dial tcp 10.133.0.132:8080: connect: connection refused" start-of-body= Oct 30 19:59:25.247354 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.247230 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.132:8080/ready\": dial tcp 10.133.0.132:8080: connect: connection refused" Oct 30 19:59:25.595283 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.595251 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:59:25.595736 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.595305 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:25.653486 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.653415 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" podStartSLOduration=3.653391756 podStartE2EDuration="3.653391756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:25.198996437 +0000 UTC m=+1725.266832005" watchObservedRunningTime="2026-10-30 19:59:25.653391756 +0000 UTC m=+1725.721227324" Oct 30 19:59:25.661407 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:25.661374 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr"] Oct 30 19:59:25.668248 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:25.668213 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac165ad_77dd_46a3_ac20_ebc4aa41c849.slice/crio-00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024 WatchSource:0}: Error finding container 00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024: Status 404 returned error can't find the container with id 00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024 Oct 30 19:59:26.280606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:26.280416 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:26.292999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:26.290354 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" event={"ID":"9ac165ad-77dd-46a3-ac20-ebc4aa41c849","Type":"ContainerStarted","Data":"00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024"} Oct 30 19:59:26.292999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:26.292604 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-g9dvq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:59:26.292999 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:26.292668 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:59:26.451695 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:26.451603 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.092738 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:27.091243 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 19:59:27.335246 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:27.335208 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" event={"ID":"9ac165ad-77dd-46a3-ac20-ebc4aa41c849","Type":"ContainerStarted","Data":"23eb08a48417c97e4318fa5dcf9ef634cf0d3a82239f2032759fbda38dbcdb1f"} Oct 30 19:59:27.990828 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:27.990797 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 19:59:27.991032 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:27.990788 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.991844 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:27.991817 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 19:59:28.357138 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:28.357097 5733 generic.go:356] "Generic (PLEG): container finished" podID="9ac165ad-77dd-46a3-ac20-ebc4aa41c849" containerID="23eb08a48417c97e4318fa5dcf9ef634cf0d3a82239f2032759fbda38dbcdb1f" exitCode=0 Oct 30 19:59:28.357564 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:28.357206 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" event={"ID":"9ac165ad-77dd-46a3-ac20-ebc4aa41c849","Type":"ContainerDied","Data":"23eb08a48417c97e4318fa5dcf9ef634cf0d3a82239f2032759fbda38dbcdb1f"} Oct 30 19:59:28.952272 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:28.952235 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.368820 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.362091 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 19:59:29.421685 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:59:29.421651 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:59:29.495143 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.495094 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:29.630543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.630462 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\") pod \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " Oct 30 19:59:29.630682 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.630597 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\" (UniqueName: \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\") pod \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " Oct 30 19:59:29.631053 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.631036 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\") pod \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\" (UID: \"9ac165ad-77dd-46a3-ac20-ebc4aa41c849\") " Oct 30 19:59:29.661140 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.661096 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca" pod "9ac165ad-77dd-46a3-ac20-ebc4aa41c849" (UID: "9ac165ad-77dd-46a3-ac20-ebc4aa41c849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:29.680912 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.680874 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca" pod "9ac165ad-77dd-46a3-ac20-ebc4aa41c849" (UID: "9ac165ad-77dd-46a3-ac20-ebc4aa41c849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:29.681052 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.680936 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc" pod "9ac165ad-77dd-46a3-ac20-ebc4aa41c849" (UID: "9ac165ad-77dd-46a3-ac20-ebc4aa41c849"). InnerVolumeSpecName "kube-api-access-vnsfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:29.732938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.732899 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:29.732938 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.732936 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:29.733176 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.732952 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnsfc\" (UniqueName: \"kubernetes.io/projected/9ac165ad-77dd-46a3-ac20-ebc4aa41c849-kube-api-access-vnsfc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:29.962985 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:29.962871 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:30.439547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:30.439511 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" Oct 30 19:59:30.439547 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:30.439534 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr" event={"ID":"9ac165ad-77dd-46a3-ac20-ebc4aa41c849","Type":"ContainerDied","Data":"00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024"} Oct 30 19:59:30.440099 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:30.439596 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00b095d013df7d9b80d5c38bb442bc149bd839419f4ab4ebb66f3227fe572024" Oct 30 19:59:31.088668 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:31.088622 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.101443 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:31.101411 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn"] Oct 30 19:59:31.106092 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:31.106052 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92e2d96d-w7gpn"] Oct 30 19:59:31.465496 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:31.465458 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:32.056629 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.056543 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34acfa86-26f8-4511-8dc8-5060a185b612" path="/var/lib/kubelet/pods/34acfa86-26f8-4511-8dc8-5060a185b612/volumes" Oct 30 19:59:32.319705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.319606 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:32.467038 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.467005 5733 generic.go:356] "Generic (PLEG): container finished" podID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerID="6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e" exitCode=0 Oct 30 19:59:32.467477 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.467087 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerDied","Data":"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e"} Oct 30 19:59:32.467646 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.467627 5733 scope.go:122] "RemoveContainer" containerID="6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e" Oct 30 19:59:32.469877 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.469843 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:32.470000 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.469981 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:32.470198 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.470167 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 19:59:32.496515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:32.496485 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.479532 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.479390 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerStarted","Data":"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1"} Oct 30 19:59:33.498334 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.498301 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.686556 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.686507 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.884360 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.884322 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:59:33.885281 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.885247 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:33.885420 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:33.885305 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:34.393071 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.393032 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 19:59:34.453920 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.453886 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 19:59:34.571101 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.568464 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.769070 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.768568 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.911093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.896557 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:34.911093 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:34.896682 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:35.583902 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:35.583853 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.786091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:35.784584 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.884705 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:35.884669 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:35.884882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:35.884725 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:35.936491 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:35.936453 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.792476 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:36.792441 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.885218 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:36.885178 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:36.885438 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:36.885235 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:37.727181 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:37.722353 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:37.884808 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:37.884778 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:37.885238 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:37.884830 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:38.692090 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:38.691718 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.884756 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:38.884721 5733 patch_prober.go:29] interesting pod/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 19:59:38.884937 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:38.884777 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 19:59:39.393952 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.393911 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" containerID="cri-o://e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72" gracePeriod=25 Oct 30 19:59:39.403996 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.403965 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9"] Oct 30 19:59:39.404886 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.404868 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9ac165ad-77dd-46a3-ac20-ebc4aa41c849" containerName="revoke-keys" Oct 30 19:59:39.404956 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.404894 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ac165ad-77dd-46a3-ac20-ebc4aa41c849" containerName="revoke-keys" Oct 30 19:59:39.405044 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.405034 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ac165ad-77dd-46a3-ac20-ebc4aa41c849" containerName="revoke-keys" Oct 30 19:59:39.407979 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.407958 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.418108 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.415675 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:59:39.418108 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.415775 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 19:59:39.424338 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.423073 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9"] Oct 30 19:59:39.454915 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.454868 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" containerID="cri-o://bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09" gracePeriod=25 Oct 30 19:59:39.558801 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.558769 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.559035 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.559017 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.559147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.559129 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wd66\" (UniqueName: \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.594605 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.594573 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:59:39.660762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.660680 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.660945 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.660771 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wd66\" (UniqueName: \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.665606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.661019 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.665606 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.662699 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.666872 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.666820 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.683515 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.683258 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:39.695986 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.695695 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wd66\" (UniqueName: \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\") pod \"maas-api-revoke-sub-ad4a5e99-fl7s9\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.740036 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.739337 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:39.889882 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.889845 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:59:39.891173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:39.891147 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 19:59:40.054715 ip-10-0-139-103 kubenswrapper[5733]: W1030 19:59:40.054659 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c13f63_f778_4d85_9f09_d61e80ad7ad1.slice/crio-c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74 WatchSource:0}: Error finding container c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74: Status 404 returned error can't find the container with id c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74 Oct 30 19:59:40.071445 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:40.071404 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9"] Oct 30 19:59:40.754815 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:40.754772 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.809388 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:40.809340 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" event={"ID":"a8c13f63-f778-4d85-9f09-d61e80ad7ad1","Type":"ContainerStarted","Data":"4727b7d6ef7a4c0ee51e6f9c5403ca2450b0bd0756b75544dedfac5fe7350128"} Oct 30 19:59:40.809588 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:40.809435 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" event={"ID":"a8c13f63-f778-4d85-9f09-d61e80ad7ad1","Type":"ContainerStarted","Data":"c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74"} Oct 30 19:59:40.836327 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:40.832854 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" podStartSLOduration=1.832811753 podStartE2EDuration="1.832811753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:40.830724386 +0000 UTC m=+1740.898559956" watchObservedRunningTime="2026-10-30 19:59:40.832811753 +0000 UTC m=+1740.900647354" Oct 30 19:59:41.805571 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:41.805535 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:41.845574 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:41.845539 5733 generic.go:356] "Generic (PLEG): container finished" podID="a8c13f63-f778-4d85-9f09-d61e80ad7ad1" containerID="4727b7d6ef7a4c0ee51e6f9c5403ca2450b0bd0756b75544dedfac5fe7350128" exitCode=0 Oct 30 19:59:41.845972 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:41.845943 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" event={"ID":"a8c13f63-f778-4d85-9f09-d61e80ad7ad1","Type":"ContainerDied","Data":"4727b7d6ef7a4c0ee51e6f9c5403ca2450b0bd0756b75544dedfac5fe7350128"} Oct 30 19:59:42.633964 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:42.633905 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:43.068447 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.068422 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:43.159144 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.159104 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\") pod \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " Oct 30 19:59:43.159342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.159159 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\") pod \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " Oct 30 19:59:43.159342 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.159211 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\" (UniqueName: \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\") pod \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\" (UID: \"a8c13f63-f778-4d85-9f09-d61e80ad7ad1\") " Oct 30 19:59:43.162428 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.162397 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca" pod "a8c13f63-f778-4d85-9f09-d61e80ad7ad1" (UID: "a8c13f63-f778-4d85-9f09-d61e80ad7ad1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:43.163493 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.163464 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca" pod "a8c13f63-f778-4d85-9f09-d61e80ad7ad1" (UID: "a8c13f63-f778-4d85-9f09-d61e80ad7ad1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:43.163658 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.163633 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66" pod "a8c13f63-f778-4d85-9f09-d61e80ad7ad1" (UID: "a8c13f63-f778-4d85-9f09-d61e80ad7ad1"). InnerVolumeSpecName "kube-api-access-7wd66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:43.227631 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.227594 5733 scope.go:122] "RemoveContainer" containerID="3ffbd318996035f213088e8374430869cb054500e07a3056cc2da16ff99ecbe5" Oct 30 19:59:43.246832 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.246500 5733 scope.go:122] "RemoveContainer" containerID="33e58c6ce46c2f9d9706f08d23b20cd4d59a426267d72b3030a03c0bb5e68a28" Oct 30 19:59:43.265992 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.263862 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:43.265992 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.263899 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:43.265992 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.263914 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wd66\" (UniqueName: \"kubernetes.io/projected/a8c13f63-f778-4d85-9f09-d61e80ad7ad1-kube-api-access-7wd66\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:43.641082 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.640657 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:43.868798 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.868769 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" Oct 30 19:59:43.868976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.868757 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9" event={"ID":"a8c13f63-f778-4d85-9f09-d61e80ad7ad1","Type":"ContainerDied","Data":"c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74"} Oct 30 19:59:43.868976 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:43.868904 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c55f6870e382183872478314fe20eff0788a752c1b45ebe368dc134e6986eb74" Oct 30 19:59:44.462080 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:44.461236 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.341147 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:45.337735 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.952199 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:45.952136 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.157318 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:46.157282 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.966977 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:46.966945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.991044 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:46.990995 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:47.191377 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:47.191347 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:47.813770 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:47.813735 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:47.998909 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:47.998864 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:48.145654 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:48.145623 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:48.881726 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:48.881688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.023762 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:49.023723 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.923810 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:49.923770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.225543 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:50.225453 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.685148 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:50.685014 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.863084 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:50.855513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.639572 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:51.639530 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.843518 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:51.843481 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.007362 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:52.007277 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.189651 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:52.189598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.636352 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:52.636313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.867076 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:52.863863 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:53.744984 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:53.744945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.414106 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:55.413507 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.762803 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:55.762696 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:56.937619 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:56.937567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:57.025081 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:57.021155 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:57.539433 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:57.537774 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:59:58.011079 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:58.010718 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:59:58.035507 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:58.035474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.008745 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.008706 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.362625 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.362592 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.393267 ip-10-0-139-103 kubenswrapper[5733]: E1030 19:59:59.393230 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 19:59:59.840385 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.840013 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 19:59:59.916288 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.916254 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 19:59:59.929602 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.929569 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\") pod \"1a7473b6-ebba-4d4e-a552-a867d27fef42\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " Oct 30 19:59:59.930091 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.930034 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\" (UniqueName: \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\") pod \"1a7473b6-ebba-4d4e-a552-a867d27fef42\" (UID: \"1a7473b6-ebba-4d4e-a552-a867d27fef42\") " Oct 30 19:59:59.931167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.930415 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume" pod "1a7473b6-ebba-4d4e-a552-a867d27fef42" (UID: "1a7473b6-ebba-4d4e-a552-a867d27fef42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:59.931167 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.931123 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a7473b6-ebba-4d4e-a552-a867d27fef42-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:59.934237 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.934171 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464" pod "1a7473b6-ebba-4d4e-a552-a867d27fef42" (UID: "1a7473b6-ebba-4d4e-a552-a867d27fef42"). InnerVolumeSpecName "kube-api-access-pk464". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:59.959724 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.959691 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 19:59:59.960940 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.960917 5733 cpu_manager.go:414] "containerMap: removing container" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" Oct 30 19:59:59.961011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.960942 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" Oct 30 19:59:59.961011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.960986 5733 cpu_manager.go:414] "containerMap: removing container" podUID="a8c13f63-f778-4d85-9f09-d61e80ad7ad1" containerName="revoke-keys" Oct 30 19:59:59.961011 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.960994 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c13f63-f778-4d85-9f09-d61e80ad7ad1" containerName="revoke-keys" Oct 30 19:59:59.961173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.961041 5733 cpu_manager.go:414] "containerMap: removing container" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" Oct 30 19:59:59.961173 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.961049 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" Oct 30 19:59:59.961284 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.961270 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerName="payload-processing" Oct 30 19:59:59.961329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.961290 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c13f63-f778-4d85-9f09-d61e80ad7ad1" containerName="revoke-keys" Oct 30 19:59:59.961329 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.961306 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerName="payload-pre-processing" Oct 30 19:59:59.964273 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.964254 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 19:59:59.984905 ip-10-0-139-103 kubenswrapper[5733]: I1030 19:59:59.984849 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 20:00:00.032963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.032925 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\" (UniqueName: \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\") pod \"3341ff20-d8d6-4a10-be1a-634df0c110cc\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " Oct 30 20:00:00.033237 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.033215 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\") pod \"3341ff20-d8d6-4a10-be1a-634df0c110cc\" (UID: \"3341ff20-d8d6-4a10-be1a-634df0c110cc\") " Oct 30 20:00:00.033734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.033703 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume" pod "3341ff20-d8d6-4a10-be1a-634df0c110cc" (UID: "3341ff20-d8d6-4a10-be1a-634df0c110cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:00.034316 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.034288 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk464\" (UniqueName: \"kubernetes.io/projected/1a7473b6-ebba-4d4e-a552-a867d27fef42-kube-api-access-pk464\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.034476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.034461 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3341ff20-d8d6-4a10-be1a-634df0c110cc-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.036962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.036934 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz" pod "3341ff20-d8d6-4a10-be1a-634df0c110cc" (UID: "3341ff20-d8d6-4a10-be1a-634df0c110cc"). InnerVolumeSpecName "kube-api-access-b4rxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:00.061018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.060992 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-35deec-openshift-default-dockercfg-fqkcq\"" Oct 30 20:00:00.136754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.136714 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.136987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.136881 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjbr\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.136987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.136963 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.137522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137139 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.137522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137366 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.137522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137416 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.137522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137471 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.137522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137511 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.138032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137553 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.138032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.137645 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4rxz\" (UniqueName: \"kubernetes.io/projected/3341ff20-d8d6-4a10-be1a-634df0c110cc-kube-api-access-b4rxz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:00.192906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.192832 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:00.216325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.216268 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.226564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.226510 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:00.239377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.239348 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.239581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.239404 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.239775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.239754 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.239860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.239830 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.240008 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.239877 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.240348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.240322 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.240581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.240560 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjbr\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.240762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.240745 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.241136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.241114 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.249644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.241576 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.254647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.249763 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.254647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.250397 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.254988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.254954 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.261583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.254995 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.261753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.255250 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.261865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.255479 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.266965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.266939 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjbr\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.274895 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.274871 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\") pod \"e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.279938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.279870 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:00.305855 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.305829 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:00:00.343927 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.343883 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.344157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.343984 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw7tk\" (UniqueName: \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.420617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.408160 5733 generic.go:356] "Generic (PLEG): container finished" podID="3341ff20-d8d6-4a10-be1a-634df0c110cc" containerID="bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09" exitCode=0 Oct 30 20:00:00.420617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.408504 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" event={"ID":"3341ff20-d8d6-4a10-be1a-634df0c110cc","Type":"ContainerDied","Data":"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09"} Oct 30 20:00:00.420617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.408565 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" event={"ID":"3341ff20-d8d6-4a10-be1a-634df0c110cc","Type":"ContainerDied","Data":"2269dbe7f1c3530a6d98f939d7d7735aa52960c6e9be8458d55161a19b28a5db"} Oct 30 20:00:00.420617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.408597 5733 scope.go:122] "RemoveContainer" containerID="bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09" Oct 30 20:00:00.420617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.408780 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs" Oct 30 20:00:00.481395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.481322 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.481724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.481701 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw7tk\" (UniqueName: \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.483397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.483371 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.485796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.485772 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.510920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.510439 5733 generic.go:356] "Generic (PLEG): container finished" podID="1a7473b6-ebba-4d4e-a552-a867d27fef42" containerID="e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72" exitCode=0 Oct 30 20:00:00.510920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.510617 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" event={"ID":"1a7473b6-ebba-4d4e-a552-a867d27fef42","Type":"ContainerDied","Data":"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72"} Oct 30 20:00:00.510920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.510724 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" event={"ID":"1a7473b6-ebba-4d4e-a552-a867d27fef42","Type":"ContainerDied","Data":"fb8da1312654e0e7b1f7ac78ce675d5a8dbdff832ea7868b3af156c9d2d62ea6"} Oct 30 20:00:00.512501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.510891 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb" Oct 30 20:00:00.520350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.518531 5733 scope.go:122] "RemoveContainer" containerID="bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09" Oct 30 20:00:00.520350 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:00.518917 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09\": container with ID starting with bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09 not found: ID does not exist" containerID="bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09" Oct 30 20:00:00.520350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.518953 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09"} err="failed to get container status \"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09\": rpc error: code = NotFound desc = could not find container \"bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09\": container with ID starting with bb5318e34721932966ac83329708c17320e92d2dedb742f76f608cc2c45d6b09 not found: ID does not exist" Oct 30 20:00:00.520350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.518976 5733 scope.go:122] "RemoveContainer" containerID="e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72" Oct 30 20:00:00.529089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.528799 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 20:00:00.533702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.533246 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw7tk\" (UniqueName: \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\") pod \"maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.561356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.561304 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-xg2cs"] Oct 30 20:00:00.564752 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.564552 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:00.587511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.587445 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 20:00:00.596114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.596084 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-z4phb"] Oct 30 20:00:00.601314 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.601292 5733 scope.go:122] "RemoveContainer" containerID="e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72" Oct 30 20:00:00.601776 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:00.601693 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72\": container with ID starting with e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72 not found: ID does not exist" containerID="e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72" Oct 30 20:00:00.601776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.601733 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72"} err="failed to get container status \"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72\": rpc error: code = NotFound desc = could not find container \"e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72\": container with ID starting with e615acf1f88ba65a6fc3a5a24c3e99fa995a2d767465b8180fb3fad722f8ba72 not found: ID does not exist" Oct 30 20:00:00.626996 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:00.626950 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5797ce7_d6ba_4ca6_9afd_4692c1333161.slice/crio-9bee9c18dab650ef30c1d449cbf4f4dc3af8b8025ba5d419bd8a35ee7680802d WatchSource:0}: Error finding container 9bee9c18dab650ef30c1d449cbf4f4dc3af8b8025ba5d419bd8a35ee7680802d: Status 404 returned error can't find the container with id 9bee9c18dab650ef30c1d449cbf4f4dc3af8b8025ba5d419bd8a35ee7680802d Oct 30 20:00:00.628402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.628372 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 20:00:00.631033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.630997 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:00.631143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.631103 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:00.631213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.631150 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:00.922376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:00.922347 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:01.535450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.535346 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerStarted","Data":"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a"} Oct 30 20:00:01.536011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.535986 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerStarted","Data":"9bee9c18dab650ef30c1d449cbf4f4dc3af8b8025ba5d419bd8a35ee7680802d"} Oct 30 20:00:01.537599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.537572 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerStarted","Data":"9b697f6bafc0f8e01fe848b3224a2d547bd66ab17e6f3c7550180f7034042f34"} Oct 30 20:00:01.537729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.537632 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerStarted","Data":"361f6ef656fbbd13a3dd8990cec8f5697bf5551efa4aa3e1d72a915aa3998bdf"} Oct 30 20:00:01.569869 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.569815 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podStartSLOduration=11.569801156 podStartE2EDuration="11.569801156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.564585339 +0000 UTC m=+1761.632420908" watchObservedRunningTime="2026-10-30 20:00:01.569801156 +0000 UTC m=+1761.637636724" Oct 30 20:00:01.590835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.590797 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:01.592647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:01.592601 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" podStartSLOduration=1.5925882599999999 podStartE2EDuration="1.59258826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:01.588940624 +0000 UTC m=+1761.656776193" watchObservedRunningTime="2026-10-30 20:00:01.59258826 +0000 UTC m=+1761.660423828" Oct 30 20:00:02.054486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.054450 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7473b6-ebba-4d4e-a552-a867d27fef42" path="/var/lib/kubelet/pods/1a7473b6-ebba-4d4e-a552-a867d27fef42/volumes" Oct 30 20:00:02.055035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.055017 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3341ff20-d8d6-4a10-be1a-634df0c110cc" path="/var/lib/kubelet/pods/3341ff20-d8d6-4a10-be1a-634df0c110cc/volumes" Oct 30 20:00:02.281009 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.280977 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:02.289022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.288989 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:02.332755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.332661 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:02.570676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.570640 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:02.691278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.688594 5733 generic.go:356] "Generic (PLEG): container finished" podID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerID="9b697f6bafc0f8e01fe848b3224a2d547bd66ab17e6f3c7550180f7034042f34" exitCode=60 Oct 30 20:00:02.691278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.689990 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerDied","Data":"9b697f6bafc0f8e01fe848b3224a2d547bd66ab17e6f3c7550180f7034042f34"} Oct 30 20:00:02.692620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.692596 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:02.697792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.697765 5733 scope.go:122] "RemoveContainer" containerID="9b697f6bafc0f8e01fe848b3224a2d547bd66ab17e6f3c7550180f7034042f34" Oct 30 20:00:02.709797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.709767 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:00:02.784476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:02.784431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.145041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:03.145010 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr"] Oct 30 20:00:03.165614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:03.165204 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ae43d70e-5xbdr"] Oct 30 20:00:03.455952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:03.455888 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.701379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:03.701331 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerStarted","Data":"0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9"} Oct 30 20:00:04.055881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.055802 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ac165ad-77dd-46a3-ac20-ebc4aa41c849" path="/var/lib/kubelet/pods/9ac165ad-77dd-46a3-ac20-ebc4aa41c849/volumes" Oct 30 20:00:04.151432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.151379 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.413154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.413116 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.609471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.609424 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.714111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.713999 5733 generic.go:356] "Generic (PLEG): container finished" podID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerID="0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9" exitCode=60 Oct 30 20:00:04.714111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.714086 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerDied","Data":"0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9"} Oct 30 20:00:04.714601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.714155 5733 scope.go:122] "RemoveContainer" containerID="9b697f6bafc0f8e01fe848b3224a2d547bd66ab17e6f3c7550180f7034042f34" Oct 30 20:00:04.714601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.714487 5733 scope.go:122] "RemoveContainer" containerID="0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9" Oct 30 20:00:04.714851 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:04.714831 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk_odh-ai-gateway-infra(73c8aff4-864b-4402-add4-b3cbe146bdc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" Oct 30 20:00:04.915093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:04.909212 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:05.730118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:05.730090 5733 scope.go:122] "RemoveContainer" containerID="0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9" Oct 30 20:00:05.730510 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:05.730397 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk_odh-ai-gateway-infra(73c8aff4-864b-4402-add4-b3cbe146bdc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" Oct 30 20:00:05.947139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:05.947048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:06.152085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:06.145752 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:07.044087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:07.043861 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:07.441975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:07.441739 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:08.323162 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:08.322425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:09.280374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:09.278901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.601267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:10.601215 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.909736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:10.909639 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.105989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.105954 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.481941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.481901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.595377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.595339 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 20:00:11.595568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.595542 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" containerID="cri-o://d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212" gracePeriod=30 Oct 30 20:00:11.740132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.725954 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.950282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.928332 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.950282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.946714 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 20:00:11.950282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.947114 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" containerID="cri-o://583bed1fc60a3b586e96e7a3303fa1efdfa4cdab0937eca1f6786528ecfcce52" gracePeriod=30 Oct 30 20:00:12.000846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:11.999886 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 20:00:12.000846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:12.000156 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" containerID="cri-o://9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f" gracePeriod=30 Oct 30 20:00:12.011654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:12.011615 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 20:00:12.012182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:12.012117 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" containerID="cri-o://6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706" gracePeriod=30 Oct 30 20:00:14.057220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:14.052170 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.864260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:14.864221 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.712118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:15.711384 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.877282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:15.877251 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.532087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:16.531914 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.925166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:16.925135 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 20:00:17.005149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.002165 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:17.005149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.003527 5733 cpu_manager.go:414] "containerMap: removing container" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" Oct 30 20:00:17.005149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.003548 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" Oct 30 20:00:17.005149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.003782 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerName="istio-proxy" Oct 30 20:00:17.037133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.020803 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:17.037133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.022789 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049135 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049330 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049425 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049510 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049542 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049616 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049668 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049907 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.049939 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\") pod \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\" (UID: \"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e\") " Oct 30 20:00:17.051854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.051824 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.052962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.052928 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.053312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.053119 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.053312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.053284 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.054508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054047 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.054508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054480 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.054508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054506 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.054710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054522 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.054710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054545 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.054710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.054559 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.067221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.066600 5733 generic.go:356] "Generic (PLEG): container finished" podID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" containerID="d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212" exitCode=0 Oct 30 20:00:17.067390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.067335 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:17.070000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.069384 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" Oct 30 20:00:17.070178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.070141 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.071799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.071596 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "kube-api-access-x8qng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.094885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.090443 5733 generic.go:356] "Generic (PLEG): container finished" podID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerID="583bed1fc60a3b586e96e7a3303fa1efdfa4cdab0937eca1f6786528ecfcce52" exitCode=0 Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.128498 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.128580 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" event={"ID":"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e","Type":"ContainerDied","Data":"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212"} Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.128640 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7" event={"ID":"52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e","Type":"ContainerDied","Data":"6ae3cc859c616c5ba4bf280861482943073c800db7f20459e294d4ce1eb34788"} Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.128678 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" event={"ID":"8878558a-bcd4-478e-a62e-3edcd26ab88f","Type":"ContainerDied","Data":"583bed1fc60a3b586e96e7a3303fa1efdfa4cdab0937eca1f6786528ecfcce52"} Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.128719 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.129366 5733 scope.go:122] "RemoveContainer" containerID="d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212" Oct 30 20:00:17.133146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.129405 5733 scope.go:122] "RemoveContainer" containerID="0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9" Oct 30 20:00:17.141788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.141742 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:17.148812 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.144245 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.153885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.153833 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.154142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.154121 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:17.158663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.158603 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy" pod "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" (UID: "52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.168171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.168130 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:17.172707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.172685 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.180281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.179880 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fvgx\" (UniqueName: \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.183884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.183826 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.184069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.184034 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.184139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.184075 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qng\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-kube-api-access-x8qng\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.184139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.184092 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.184139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.184108 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.192265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.192116 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.192436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.191965 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx"] Oct 30 20:00:17.201608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.200273 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:17.201608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.200369 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.208139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.206689 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:00:17.221524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.221492 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:17.222401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.222376 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:17.222584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.222568 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:17.222712 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.222699 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:17.222914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.222897 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.223511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.223477 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.223746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.223485 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx"] Oct 30 20:00:17.229909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.228619 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-35deec-dockercfg-cj5rx\"" Oct 30 20:00:17.229909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.229306 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-35deec\"" Oct 30 20:00:17.236552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.236488 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:17.243936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.243872 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:17.253950 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.253881 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:17.275182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.274170 5733 scope.go:122] "RemoveContainer" containerID="d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212" Oct 30 20:00:17.275182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.275044 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:00:17.276011 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:17.275934 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212\": container with ID starting with d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212 not found: ID does not exist" containerID="d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212" Oct 30 20:00:17.276011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.275971 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212"} err="failed to get container status \"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212\": rpc error: code = NotFound desc = could not find container \"d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212\": container with ID starting with d9c95364a0d7656f6c18222d42db31b9fe2e008c8a7e6a8e8f44d8b7309a5212 not found: ID does not exist" Oct 30 20:00:17.282788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.282696 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285637 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws4wm\" (UniqueName: \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285701 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285747 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm6rv\" (UniqueName: \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285791 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285825 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285901 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fvgx\" (UniqueName: \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.285940 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqbh\" (UniqueName: \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.286078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.286000 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxrx2\" (UniqueName: \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290003 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290092 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnx8n\" (UniqueName: \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290182 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290250 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtdp\" (UniqueName: \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290306 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290352 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsmfs\" (UniqueName: \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290387 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290422 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290512 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290540 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwk7\" (UniqueName: \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.291113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.290590 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.294803 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.294478 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.396249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396208 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws4wm\" (UniqueName: \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.396413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396291 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.396413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396342 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm6rv\" (UniqueName: \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.396413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396394 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.396600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396432 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.396600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396526 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqbh\" (UniqueName: \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.396600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396588 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxrx2\" (UniqueName: \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.396773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396692 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnx8n\" (UniqueName: \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.396828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396777 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.396926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396843 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtdp\" (UniqueName: \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.396926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396884 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.397043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396930 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsmfs\" (UniqueName: \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.397043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.396971 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.397043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.397010 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.397228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.397116 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.397228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.397148 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwk7\" (UniqueName: \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.397339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.397241 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.399323 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.399863 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.400548 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.400839 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.401436 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.401537 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.403796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.402184 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.407449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.407422 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.412668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.412583 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fvgx\" (UniqueName: \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\") pod \"maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.416942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.416882 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.417427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.417384 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnx8n\" (UniqueName: \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\") pod \"payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.418348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.418301 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxrx2\" (UniqueName: \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.425237 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.424219 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws4wm\" (UniqueName: \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.429172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.428809 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm6rv\" (UniqueName: \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\") pod \"maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.431180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.430964 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:17.433902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.433845 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtdp\" (UniqueName: \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\") pod \"maas-api-key-cleanup-29889840-b5dzm\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.434631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.434096 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqbh\" (UniqueName: \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\") pod \"maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.438417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.438369 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsmfs\" (UniqueName: \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.439323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.439294 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwk7\" (UniqueName: \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\") pod \"maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.440204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.439928 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:17.459500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.459124 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 20:00:17.464315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.464282 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 20:00:17.473317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.473285 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-gvdc7"] Oct 30 20:00:17.499556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.499523 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.499647 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.499694 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.499792 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.499853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.500038 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.500122 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.500222 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.504110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.500276 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\") pod \"70ecb511-0de4-455d-b9f8-9faea4093ff8\" (UID: \"70ecb511-0de4-455d-b9f8-9faea4093ff8\") " Oct 30 20:00:17.512905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.512864 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.538104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.524032 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.538104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.526240 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.539724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.539674 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.568775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.558308 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.568775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.565083 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.581694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.581308 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:17.587081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.586870 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "kube-api-access-zs2wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.595873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.594694 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.609138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.606522 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.609138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.606583 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:00:17.609138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.607930 5733 cpu_manager.go:414] "containerMap: removing container" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" Oct 30 20:00:17.609138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.607947 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" Oct 30 20:00:17.609138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.608190 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerName="istio-proxy" Oct 30 20:00:17.616365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.616197 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo" pod "70ecb511-0de4-455d-b9f8-9faea4093ff8" (UID: "70ecb511-0de4-455d-b9f8-9faea4093ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:17.620946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.620874 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:17.622245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.622189 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.622469 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623513 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs2wj\" (UniqueName: \"kubernetes.io/projected/70ecb511-0de4-455d-b9f8-9faea4093ff8-kube-api-access-zs2wj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623533 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623548 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623562 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ecb511-0de4-455d-b9f8-9faea4093ff8-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623576 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623589 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623604 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.623655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.623618 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ecb511-0de4-455d-b9f8-9faea4093ff8-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.624743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.624525 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z"] Oct 30 20:00:17.628363 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.628312 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:17.630071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.630036 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 20:00:17.650371 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.650290 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:17.650699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.650670 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:00:17.650829 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.650815 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.652945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.652895 5733 cpu_manager.go:414] "containerMap: removing container" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" Oct 30 20:00:17.653149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.653135 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" Oct 30 20:00:17.658076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.658003 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerName="istio-proxy" Oct 30 20:00:17.721948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.705853 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:17.721948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.712896 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:17.721948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.715361 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:17.724630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724249 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.724630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724381 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.724630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724431 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.724630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724499 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.724630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724552 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724631 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724676 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724804 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724836 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724861 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.725043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.724917 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\") pod \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\" (UID: \"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0\") " Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.725489 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.725581 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmrz7\" (UniqueName: \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.725746 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.725852 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.726066 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj4tj\" (UniqueName: \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.726217 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.726396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.726238 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.726958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.726673 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:17.734398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.734370 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.734704 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.734688 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.738011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.736442 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z"] Oct 30 20:00:17.738011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.736492 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:00:17.738011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.736930 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:17.747646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.745898 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae43d70e\"" Oct 30 20:00:17.747646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.746035 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae43d70e-dockercfg-zbhdk\"" Oct 30 20:00:17.748499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.748442 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:00:17.748698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.748668 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.748873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.748848 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "kube-api-access-5b4hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:17.755072 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.753763 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:17.762005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.761922 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy" pod "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" (UID: "650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.863655 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj4tj\" (UniqueName: \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.863837 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.863940 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmrz7\" (UniqueName: \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.864171 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.864305 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.865535 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.865912 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.870042 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:17.880603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.876696 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.924871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.924837 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.925174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.925155 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.925943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.925923 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b4hp\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-kube-api-access-5b4hp\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.926081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.926024 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.926209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.926196 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.926357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.926344 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:17.956736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.956663 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj4tj\" (UniqueName: \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\") pod \"maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:17.964575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.963376 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:00:17.994108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:17.994028 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmrz7\" (UniqueName: \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\") pod \"payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:18.013041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.012948 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.017115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.016849 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:18.028859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.028807 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzdhw\" (UniqueName: \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.028978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.028964 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.044620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.031760 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:18.044620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.035643 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:00:18.076378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.076345 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e" path="/var/lib/kubelet/pods/52fd6a8d-27a4-40c4-b7db-7d7e0fccd46e/volumes" Oct 30 20:00:18.134763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.130471 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzdhw\" (UniqueName: \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.134763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.130701 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.134763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.130985 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2js25\" (UniqueName: \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.134763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.131156 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.149764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.146117 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.157026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.155280 5733 generic.go:356] "Generic (PLEG): container finished" podID="70ecb511-0de4-455d-b9f8-9faea4093ff8" containerID="6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706" exitCode=0 Oct 30 20:00:18.157026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.155447 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" event={"ID":"70ecb511-0de4-455d-b9f8-9faea4093ff8","Type":"ContainerDied","Data":"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706"} Oct 30 20:00:18.157026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.155509 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" event={"ID":"70ecb511-0de4-455d-b9f8-9faea4093ff8","Type":"ContainerDied","Data":"595e169146e7a9c5c5fb30655b79e9b402355af396ef5491fa3891fd64f96d91"} Oct 30 20:00:18.157026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.155542 5733 scope.go:122] "RemoveContainer" containerID="6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706" Oct 30 20:00:18.157026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.155727 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb" Oct 30 20:00:18.183414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.181374 5733 generic.go:356] "Generic (PLEG): container finished" podID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" containerID="9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f" exitCode=0 Oct 30 20:00:18.183414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.181576 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" event={"ID":"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0","Type":"ContainerDied","Data":"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f"} Oct 30 20:00:18.183414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.181626 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" event={"ID":"650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0","Type":"ContainerDied","Data":"052d1f9009159756835233253f626c82916ace65b505533654729362e62c1201"} Oct 30 20:00:18.183414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.181744 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl" Oct 30 20:00:18.194307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.194277 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzdhw\" (UniqueName: \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\") pod \"payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.199125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.198521 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerStarted","Data":"ce85432a0e73eb32207bdb9755a0fe37c3a5f12b488d37b9858640e691f6a9ab"} Oct 30 20:00:18.244356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.240404 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2js25\" (UniqueName: \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.244356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.240575 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.244356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.242237 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.265401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.265230 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 20:00:18.273939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.273904 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-gw6wb"] Oct 30 20:00:18.301483 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.300487 5733 scope.go:122] "RemoveContainer" containerID="6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706" Oct 30 20:00:18.301483 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.301009 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 20:00:18.313674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.310958 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-l54kl"] Oct 30 20:00:18.313674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.312801 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2js25\" (UniqueName: \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\") pod \"payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.314743 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:18.314276 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706\": container with ID starting with 6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706 not found: ID does not exist" containerID="6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706" Oct 30 20:00:18.314743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.314313 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706"} err="failed to get container status \"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706\": rpc error: code = NotFound desc = could not find container \"6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706\": container with ID starting with 6ff6c1c52628ff9527aee3fc6e2eb5eaeff15435cdb75c9ece85980df628e706 not found: ID does not exist" Oct 30 20:00:18.314743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.314339 5733 scope.go:122] "RemoveContainer" containerID="9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f" Oct 30 20:00:18.327698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.327453 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:18.369538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.369455 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:18.371571 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:18.371537 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b553ede_8241_48cb_9d9c_fcb7df9f0a90.slice/crio-f58a558dcb263242f65069bf3de993cfdc813fb39660b4e501811da84601891a WatchSource:0}: Error finding container f58a558dcb263242f65069bf3de993cfdc813fb39660b4e501811da84601891a: Status 404 returned error can't find the container with id f58a558dcb263242f65069bf3de993cfdc813fb39660b4e501811da84601891a Oct 30 20:00:18.378880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.378832 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:18.386405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.386289 5733 scope.go:122] "RemoveContainer" containerID="9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f" Oct 30 20:00:18.387361 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:18.387267 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f\": container with ID starting with 9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f not found: ID does not exist" containerID="9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f" Oct 30 20:00:18.387361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.387309 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f"} err="failed to get container status \"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f\": rpc error: code = NotFound desc = could not find container \"9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f\": container with ID starting with 9771fdbfef83f6cc442853b6deddd64585af0c701eacf82839c7b5bddcbec70f not found: ID does not exist" Oct 30 20:00:18.397457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.397219 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:18.464152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.464106 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.597179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.597155 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665618 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665678 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665748 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665782 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665847 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665867 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665892 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.665993 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.667286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.666030 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\") pod \"8878558a-bcd4-478e-a62e-3edcd26ab88f\" (UID: \"8878558a-bcd4-478e-a62e-3edcd26ab88f\") " Oct 30 20:00:18.669597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.669498 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:18.673580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.672141 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.673580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.672349 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.673580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.673459 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.673580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.673525 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.677902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.677860 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "kube-api-access-4x98l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.678465 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.678436 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.693589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.693555 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:18.695471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.695429 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo" pod "8878558a-bcd4-478e-a62e-3edcd26ab88f" (UID: "8878558a-bcd4-478e-a62e-3edcd26ab88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:00:18.770652 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770614 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770652 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770652 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770667 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770678 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770687 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8878558a-bcd4-478e-a62e-3edcd26ab88f-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770698 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770706 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x98l\" (UniqueName: \"kubernetes.io/projected/8878558a-bcd4-478e-a62e-3edcd26ab88f-kube-api-access-4x98l\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770717 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8878558a-bcd4-478e-a62e-3edcd26ab88f-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.770846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.770726 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8878558a-bcd4-478e-a62e-3edcd26ab88f-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.783455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.783426 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:18.833749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.833719 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:00:18.924276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.924246 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:18.969158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:18.967819 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:00:19.108730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.102179 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:19.127782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.127657 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:19.147216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.146170 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:00:19.147216 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:19.146186 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16214681_4e48_4e56_bc49_3a9511b2ea66.slice/crio-8cbbe74e437aaa98e6de8383c21f36ebde2687cd5339127729fb6b510386abc1 WatchSource:0}: Error finding container 8cbbe74e437aaa98e6de8383c21f36ebde2687cd5339127729fb6b510386abc1: Status 404 returned error can't find the container with id 8cbbe74e437aaa98e6de8383c21f36ebde2687cd5339127729fb6b510386abc1 Oct 30 20:00:19.158725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.158700 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:19.163329 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:19.163305 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1032716_d9e7_4ff9_8504_760423b63c34.slice/crio-fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93 WatchSource:0}: Error finding container fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93: Status 404 returned error can't find the container with id fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93 Oct 30 20:00:19.164876 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:19.164851 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ebb26b_cecd_46fd_a23d_368a78afdf03.slice/crio-165deadddff5825089f0f9a332354aadb1669b0d9e8fb2aff7e0919a37732fbd WatchSource:0}: Error finding container 165deadddff5825089f0f9a332354aadb1669b0d9e8fb2aff7e0919a37732fbd: Status 404 returned error can't find the container with id 165deadddff5825089f0f9a332354aadb1669b0d9e8fb2aff7e0919a37732fbd Oct 30 20:00:19.166440 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:19.166178 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5db55fe_d191_45ce_92d6_7b44dc4c68c3.slice/crio-353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9 WatchSource:0}: Error finding container 353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9: Status 404 returned error can't find the container with id 353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9 Oct 30 20:00:19.167886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.167836 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z"] Oct 30 20:00:19.177213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.173739 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx"] Oct 30 20:00:19.181723 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:19.181699 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b4aea87_5fa6_444f_bb9f_a14fcb7f3561.slice/crio-91d08f374eb209179946d62cd8d49e70af5e7af47a1b180b1222df38333985ee WatchSource:0}: Error finding container 91d08f374eb209179946d62cd8d49e70af5e7af47a1b180b1222df38333985ee: Status 404 returned error can't find the container with id 91d08f374eb209179946d62cd8d49e70af5e7af47a1b180b1222df38333985ee Oct 30 20:00:19.181963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.181940 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:00:19.224240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.224210 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerStarted","Data":"f58a558dcb263242f65069bf3de993cfdc813fb39660b4e501811da84601891a"} Oct 30 20:00:19.233991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.233959 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerStarted","Data":"f51cbcfa52202c03c8ca7ae665f1df6281e5d29c4324fbb4493c6ee67b59e329"} Oct 30 20:00:19.236032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.235999 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerStarted","Data":"0ecc27e7167b2f166273d0762d6789147f12bb73e15d01148a6499ab8a1bb8e9"} Oct 30 20:00:19.239022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.238996 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" event={"ID":"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561","Type":"ContainerStarted","Data":"91d08f374eb209179946d62cd8d49e70af5e7af47a1b180b1222df38333985ee"} Oct 30 20:00:19.241242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.240946 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerStarted","Data":"95fa492e9a05ffd4316f08fc73ce414d24da8ca401c1763b19dc479f87172ac8"} Oct 30 20:00:19.244390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.244344 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" event={"ID":"f43cb101-2fd1-4418-ac86-3920dc32477b","Type":"ContainerStarted","Data":"989985e86c6f9b4c492f8a1aaeb861709b899198d95f41ec3fedd4e5361e0d30"} Oct 30 20:00:19.244479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.244416 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" event={"ID":"f43cb101-2fd1-4418-ac86-3920dc32477b","Type":"ContainerStarted","Data":"f1ee1a39c6b6d304425905043e96d010a6962373e166e9f9be06bbdaf6dbe38e"} Oct 30 20:00:19.244479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.244465 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:19.246990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.246954 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" event={"ID":"d1032716-d9e7-4ff9-8504-760423b63c34","Type":"ContainerStarted","Data":"fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93"} Oct 30 20:00:19.258557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.258523 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerStarted","Data":"df494a82deb23e705fb77e86cdad86b546de86746b1c8b9da8991623520ec5cd"} Oct 30 20:00:19.258684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.258575 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerStarted","Data":"a8030dbfc81446789d9c95fe58890b42697fb5eb50cf50393f0581ebc9081ebd"} Oct 30 20:00:19.262036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.262007 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerStarted","Data":"f976a97a57f9ec716ae90bbe5a8edf43be861921d7c86f15a4192498ed36a4f2"} Oct 30 20:00:19.263453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.263430 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerStarted","Data":"165deadddff5825089f0f9a332354aadb1669b0d9e8fb2aff7e0919a37732fbd"} Oct 30 20:00:19.264841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.264817 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" event={"ID":"16214681-4e48-4e56-bc49-3a9511b2ea66","Type":"ContainerStarted","Data":"8cbbe74e437aaa98e6de8383c21f36ebde2687cd5339127729fb6b510386abc1"} Oct 30 20:00:19.266557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.266537 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" event={"ID":"8878558a-bcd4-478e-a62e-3edcd26ab88f","Type":"ContainerDied","Data":"1e080b077e9ecdce444c65dc904be0de8674e13456c178c28f3692489e5de98d"} Oct 30 20:00:19.266627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.266590 5733 scope.go:122] "RemoveContainer" containerID="583bed1fc60a3b586e96e7a3303fa1efdfa4cdab0937eca1f6786528ecfcce52" Oct 30 20:00:19.266792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.266775 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" Oct 30 20:00:19.273358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.273161 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" podStartSLOduration=14.273148495 podStartE2EDuration="14.273148495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:19.269386666 +0000 UTC m=+1779.337222234" watchObservedRunningTime="2026-10-30 20:00:19.273148495 +0000 UTC m=+1779.340984063" Oct 30 20:00:19.273897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.273869 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" event={"ID":"4ad346e3-51bb-493b-a76f-71a08cca6a6b","Type":"ContainerStarted","Data":"9ffd4bf2237dce2f8d6e30cd0d0c426ae276f6983fe9dd044a954f9de62a6250"} Oct 30 20:00:19.277534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.277507 5733 generic.go:356] "Generic (PLEG): container finished" podID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerID="ce85432a0e73eb32207bdb9755a0fe37c3a5f12b488d37b9858640e691f6a9ab" exitCode=60 Oct 30 20:00:19.277631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.277575 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerDied","Data":"ce85432a0e73eb32207bdb9755a0fe37c3a5f12b488d37b9858640e691f6a9ab"} Oct 30 20:00:19.278187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.278168 5733 scope.go:122] "RemoveContainer" containerID="ce85432a0e73eb32207bdb9755a0fe37c3a5f12b488d37b9858640e691f6a9ab" Oct 30 20:00:19.278616 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:19.278597 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk_odh-ai-gateway-infra(73c8aff4-864b-4402-add4-b3cbe146bdc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" Oct 30 20:00:19.279098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.279046 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" event={"ID":"b5db55fe-d191-45ce-92d6-7b44dc4c68c3","Type":"ContainerStarted","Data":"353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9"} Oct 30 20:00:19.284472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.284448 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:19.289729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.289709 5733 scope.go:122] "RemoveContainer" containerID="0c967ee268ce86649a01f72255b547e05860819e631de1248941959e9c10eca9" Oct 30 20:00:19.307564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.307516 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" podStartSLOduration=19.30750201 podStartE2EDuration="19.30750201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:19.299974769 +0000 UTC m=+1779.367810337" watchObservedRunningTime="2026-10-30 20:00:19.30750201 +0000 UTC m=+1779.375337575" Oct 30 20:00:19.375019 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.374984 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 20:00:19.386531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.386499 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66"] Oct 30 20:00:19.463403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.463361 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:00:19.463583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.463431 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-w7n66" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:00:19.486749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:19.486623 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.056353 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.056320 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0" path="/var/lib/kubelet/pods/650d8bb7-8a2b-4f82-b1b2-0fb2484d07e0/volumes" Oct 30 20:00:20.056953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.056935 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ecb511-0de4-455d-b9f8-9faea4093ff8" path="/var/lib/kubelet/pods/70ecb511-0de4-455d-b9f8-9faea4093ff8/volumes" Oct 30 20:00:20.057779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.057754 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" path="/var/lib/kubelet/pods/8878558a-bcd4-478e-a62e-3edcd26ab88f/volumes" Oct 30 20:00:20.290629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.290529 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" event={"ID":"4ad346e3-51bb-493b-a76f-71a08cca6a6b","Type":"ContainerStarted","Data":"7339e929cfcd01788e4a9b7c06a66f1390dcce9223ef2672e1bc374d3f7af6e2"} Oct 30 20:00:20.290629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.290599 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:20.294925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.294893 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" event={"ID":"b5db55fe-d191-45ce-92d6-7b44dc4c68c3","Type":"ContainerStarted","Data":"2fc2d4c247bade70f7523486168270c78ff54d771e531e9726fbc6b2d38066a4"} Oct 30 20:00:20.296968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.296940 5733 generic.go:356] "Generic (PLEG): container finished" podID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerID="c598031cb35543a6101c5faf71ef6dbceefc4cb37a8108bdf6729cf0c3c15190" exitCode=60 Oct 30 20:00:20.297123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.297037 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerDied","Data":"c598031cb35543a6101c5faf71ef6dbceefc4cb37a8108bdf6729cf0c3c15190"} Oct 30 20:00:20.297279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.297263 5733 scope.go:122] "RemoveContainer" containerID="c598031cb35543a6101c5faf71ef6dbceefc4cb37a8108bdf6729cf0c3c15190" Oct 30 20:00:20.299025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.299004 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerStarted","Data":"0337acdc42ecf68ae3e9dd4aae7bbe63dfab4a627d87af5792f27031d9540745"} Oct 30 20:00:20.301258 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.301236 5733 generic.go:356] "Generic (PLEG): container finished" podID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerID="003e624fc8dd9a1606bba949beaaaf1613d680e4f8a551bfe6ca42006fcae53d" exitCode=60 Oct 30 20:00:20.301378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.301353 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerDied","Data":"003e624fc8dd9a1606bba949beaaaf1613d680e4f8a551bfe6ca42006fcae53d"} Oct 30 20:00:20.301546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.301531 5733 scope.go:122] "RemoveContainer" containerID="003e624fc8dd9a1606bba949beaaaf1613d680e4f8a551bfe6ca42006fcae53d" Oct 30 20:00:20.305843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.305820 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" event={"ID":"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561","Type":"ContainerStarted","Data":"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00"} Oct 30 20:00:20.305956 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.305892 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:20.330522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.330454 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" podStartSLOduration=15.330435085 podStartE2EDuration="15.330435085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.319925527 +0000 UTC m=+1780.387761097" watchObservedRunningTime="2026-10-30 20:00:20.330435085 +0000 UTC m=+1780.398270655" Oct 30 20:00:20.384858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.368508 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerStarted","Data":"4f20801c481f01072679e6461d829a6645b123c3a97a78dcb7c53d4600301721"} Oct 30 20:00:20.384858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.378873 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" event={"ID":"d1032716-d9e7-4ff9-8504-760423b63c34","Type":"ContainerStarted","Data":"edc236e4a0253c53971ca13f1748f6b9d68dc537f2711204bd9de805b2ca9270"} Oct 30 20:00:20.384858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.381972 5733 generic.go:356] "Generic (PLEG): container finished" podID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerID="df494a82deb23e705fb77e86cdad86b546de86746b1c8b9da8991623520ec5cd" exitCode=60 Oct 30 20:00:20.384858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.382298 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerDied","Data":"df494a82deb23e705fb77e86cdad86b546de86746b1c8b9da8991623520ec5cd"} Oct 30 20:00:20.384858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.382570 5733 scope.go:122] "RemoveContainer" containerID="df494a82deb23e705fb77e86cdad86b546de86746b1c8b9da8991623520ec5cd" Oct 30 20:00:20.394769 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.392593 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerStarted","Data":"f41b8f5bb1aa526ad553e80ed7befe5176cf721c7690ec7dafdd098028b9a7b4"} Oct 30 20:00:20.394769 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.392963 5733 scope.go:122] "RemoveContainer" containerID="f41b8f5bb1aa526ad553e80ed7befe5176cf721c7690ec7dafdd098028b9a7b4" Oct 30 20:00:20.406576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.405969 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerStarted","Data":"cc75adc28e36a56c4d5614d90e14a892f2bd5f0e909b386c2e476de2834056ed"} Oct 30 20:00:20.421312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.421266 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" event={"ID":"16214681-4e48-4e56-bc49-3a9511b2ea66","Type":"ContainerStarted","Data":"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce"} Oct 30 20:00:20.421634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.421582 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" podStartSLOduration=14.421565071 podStartE2EDuration="14.421565071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.406815327 +0000 UTC m=+1780.474650896" watchObservedRunningTime="2026-10-30 20:00:20.421565071 +0000 UTC m=+1780.489400641" Oct 30 20:00:20.422295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.422253 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:20.484413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.483531 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" podStartSLOduration=20.48351246 podStartE2EDuration="20.48351246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.468652776 +0000 UTC m=+1780.536488342" watchObservedRunningTime="2026-10-30 20:00:20.48351246 +0000 UTC m=+1780.551348029" Oct 30 20:00:20.533864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.532556 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" podStartSLOduration=9.532535715 podStartE2EDuration="9.532535715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.523582415 +0000 UTC m=+1780.591418011" watchObservedRunningTime="2026-10-30 20:00:20.532535715 +0000 UTC m=+1780.600371300" Oct 30 20:00:20.550427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.550331 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" podStartSLOduration=20.550319387000002 podStartE2EDuration="20.550319387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.547652329 +0000 UTC m=+1780.615487898" watchObservedRunningTime="2026-10-30 20:00:20.550319387 +0000 UTC m=+1780.618154956" Oct 30 20:00:20.551332 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.551283 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:20.577275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.577213 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" podStartSLOduration=9.577194956 podStartE2EDuration="9.577194956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.574940902 +0000 UTC m=+1780.642776471" watchObservedRunningTime="2026-10-30 20:00:20.577194956 +0000 UTC m=+1780.645030526" Oct 30 20:00:20.645232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.645175 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" podStartSLOduration=20.645157019 podStartE2EDuration="20.645157019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.642753844 +0000 UTC m=+1780.710589416" watchObservedRunningTime="2026-10-30 20:00:20.645157019 +0000 UTC m=+1780.712992588" Oct 30 20:00:20.668796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.668753 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" podStartSLOduration=20.668738682 podStartE2EDuration="20.668738682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:20.664893459 +0000 UTC m=+1780.732729027" watchObservedRunningTime="2026-10-30 20:00:20.668738682 +0000 UTC m=+1780.736574249" Oct 30 20:00:20.754327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.752340 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:20.882480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.882449 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\" (UniqueName: \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\") pod \"73c8aff4-864b-4402-add4-b3cbe146bdc5\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " Oct 30 20:00:20.882655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.882618 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\") pod \"73c8aff4-864b-4402-add4-b3cbe146bdc5\" (UID: \"73c8aff4-864b-4402-add4-b3cbe146bdc5\") " Oct 30 20:00:20.882994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.882968 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca" pod "73c8aff4-864b-4402-add4-b3cbe146bdc5" (UID: "73c8aff4-864b-4402-add4-b3cbe146bdc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:20.883393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.883374 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73c8aff4-864b-4402-add4-b3cbe146bdc5-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.911469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.911433 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk" pod "73c8aff4-864b-4402-add4-b3cbe146bdc5" (UID: "73c8aff4-864b-4402-add4-b3cbe146bdc5"). InnerVolumeSpecName "kube-api-access-rw7tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.984707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:20.984670 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw7tk\" (UniqueName: \"kubernetes.io/projected/73c8aff4-864b-4402-add4-b3cbe146bdc5-kube-api-access-rw7tk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.342339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.342308 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:21.441609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.441519 5733 generic.go:356] "Generic (PLEG): container finished" podID="b5db55fe-d191-45ce-92d6-7b44dc4c68c3" containerID="2fc2d4c247bade70f7523486168270c78ff54d771e531e9726fbc6b2d38066a4" exitCode=0 Oct 30 20:00:21.441609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.441598 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" event={"ID":"b5db55fe-d191-45ce-92d6-7b44dc4c68c3","Type":"ContainerDied","Data":"2fc2d4c247bade70f7523486168270c78ff54d771e531e9726fbc6b2d38066a4"} Oct 30 20:00:21.443944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.443917 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerStarted","Data":"8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3"} Oct 30 20:00:21.445985 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.445956 5733 generic.go:356] "Generic (PLEG): container finished" podID="0c972e35-d044-4548-afa9-003279837087" containerID="0337acdc42ecf68ae3e9dd4aae7bbe63dfab4a627d87af5792f27031d9540745" exitCode=60 Oct 30 20:00:21.446153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.446046 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerDied","Data":"0337acdc42ecf68ae3e9dd4aae7bbe63dfab4a627d87af5792f27031d9540745"} Oct 30 20:00:21.446554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.446497 5733 scope.go:122] "RemoveContainer" containerID="0337acdc42ecf68ae3e9dd4aae7bbe63dfab4a627d87af5792f27031d9540745" Oct 30 20:00:21.484821 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.470573 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerStarted","Data":"d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0"} Oct 30 20:00:21.489788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.488461 5733 generic.go:356] "Generic (PLEG): container finished" podID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerID="4f20801c481f01072679e6461d829a6645b123c3a97a78dcb7c53d4600301721" exitCode=60 Oct 30 20:00:21.489788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.488568 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerDied","Data":"4f20801c481f01072679e6461d829a6645b123c3a97a78dcb7c53d4600301721"} Oct 30 20:00:21.489788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.489143 5733 scope.go:122] "RemoveContainer" containerID="4f20801c481f01072679e6461d829a6645b123c3a97a78dcb7c53d4600301721" Oct 30 20:00:21.552573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.552540 5733 generic.go:356] "Generic (PLEG): container finished" podID="d1032716-d9e7-4ff9-8504-760423b63c34" containerID="edc236e4a0253c53971ca13f1748f6b9d68dc537f2711204bd9de805b2ca9270" exitCode=0 Oct 30 20:00:21.553171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.553004 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" event={"ID":"d1032716-d9e7-4ff9-8504-760423b63c34","Type":"ContainerDied","Data":"edc236e4a0253c53971ca13f1748f6b9d68dc537f2711204bd9de805b2ca9270"} Oct 30 20:00:21.585165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.584169 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerStarted","Data":"89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd"} Oct 30 20:00:21.592078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.588403 5733 generic.go:356] "Generic (PLEG): container finished" podID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerID="f41b8f5bb1aa526ad553e80ed7befe5176cf721c7690ec7dafdd098028b9a7b4" exitCode=60 Oct 30 20:00:21.595014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.594983 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerDied","Data":"f41b8f5bb1aa526ad553e80ed7befe5176cf721c7690ec7dafdd098028b9a7b4"} Oct 30 20:00:21.595465 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.595423 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerStarted","Data":"8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b"} Oct 30 20:00:21.615288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.602727 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" podStartSLOduration=21.60271056 podStartE2EDuration="21.60271056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:21.536592363 +0000 UTC m=+1781.604427931" watchObservedRunningTime="2026-10-30 20:00:21.60271056 +0000 UTC m=+1781.670546129" Oct 30 20:00:21.676215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.661401 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:21.728227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.728190 5733 generic.go:356] "Generic (PLEG): container finished" podID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerID="cc75adc28e36a56c4d5614d90e14a892f2bd5f0e909b386c2e476de2834056ed" exitCode=60 Oct 30 20:00:21.728681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.728651 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerDied","Data":"cc75adc28e36a56c4d5614d90e14a892f2bd5f0e909b386c2e476de2834056ed"} Oct 30 20:00:21.731904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.729036 5733 scope.go:122] "RemoveContainer" containerID="cc75adc28e36a56c4d5614d90e14a892f2bd5f0e909b386c2e476de2834056ed" Oct 30 20:00:21.732547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.732523 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" event={"ID":"73c8aff4-864b-4402-add4-b3cbe146bdc5","Type":"ContainerDied","Data":"361f6ef656fbbd13a3dd8990cec8f5697bf5551efa4aa3e1d72a915aa3998bdf"} Oct 30 20:00:21.737893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.737873 5733 scope.go:122] "RemoveContainer" containerID="ce85432a0e73eb32207bdb9755a0fe37c3a5f12b488d37b9858640e691f6a9ab" Oct 30 20:00:21.738158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.735999 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk" Oct 30 20:00:21.818090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.815898 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" podStartSLOduration=21.815876003 podStartE2EDuration="21.815876003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:21.725339052 +0000 UTC m=+1781.793174621" watchObservedRunningTime="2026-10-30 20:00:21.815876003 +0000 UTC m=+1781.883711573" Oct 30 20:00:21.944467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:21.944402 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" podStartSLOduration=21.944383068 podStartE2EDuration="21.944383068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:21.944111934 +0000 UTC m=+1782.011947502" watchObservedRunningTime="2026-10-30 20:00:21.944383068 +0000 UTC m=+1782.012218638" Oct 30 20:00:22.076826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.076793 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:22.098286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.098237 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ced832-29889840-gjzgk"] Oct 30 20:00:22.749442 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.749406 5733 generic.go:356] "Generic (PLEG): container finished" podID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerID="8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3" exitCode=60 Oct 30 20:00:22.749886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.749487 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerDied","Data":"8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3"} Oct 30 20:00:22.749886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.749567 5733 scope.go:122] "RemoveContainer" containerID="c598031cb35543a6101c5faf71ef6dbceefc4cb37a8108bdf6729cf0c3c15190" Oct 30 20:00:22.750037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.750020 5733 scope.go:122] "RemoveContainer" containerID="8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3" Oct 30 20:00:22.750505 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:22.750484 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6_odh-ai-gateway-infra(6b553ede-8241-48cb-9d9c-fcb7df9f0a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" Oct 30 20:00:22.752968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.752929 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerStarted","Data":"a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c"} Oct 30 20:00:22.755108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.755079 5733 generic.go:356] "Generic (PLEG): container finished" podID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerID="d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0" exitCode=60 Oct 30 20:00:22.755231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.755201 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerDied","Data":"d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0"} Oct 30 20:00:22.755607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.755584 5733 scope.go:122] "RemoveContainer" containerID="d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0" Oct 30 20:00:22.756103 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:22.756028 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v_odh-ai-gateway-infra(e578b6ca-d719-4dcf-8d46-d59825404e14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" Oct 30 20:00:22.757458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.757433 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerStarted","Data":"846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc"} Oct 30 20:00:22.759740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.759716 5733 generic.go:356] "Generic (PLEG): container finished" podID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerID="89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd" exitCode=60 Oct 30 20:00:22.759848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.759805 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerDied","Data":"89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd"} Oct 30 20:00:22.760013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.759994 5733 scope.go:122] "RemoveContainer" containerID="89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd" Oct 30 20:00:22.760407 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:22.760386 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j_odh-ai-gateway-infra(9b5b7900-2d3a-44bc-a3e4-08b47b4c98db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" Oct 30 20:00:22.762807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.761778 5733 generic.go:356] "Generic (PLEG): container finished" podID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerID="8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b" exitCode=60 Oct 30 20:00:22.762807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.761815 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerDied","Data":"8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b"} Oct 30 20:00:22.762807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.762124 5733 scope.go:122] "RemoveContainer" containerID="8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b" Oct 30 20:00:22.762807 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:22.762413 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9_odh-ai-gateway-infra(257ce18b-20a1-4cd6-9745-47d82cc229ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" Oct 30 20:00:22.764443 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.764421 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerStarted","Data":"15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14"} Oct 30 20:00:22.770196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.770176 5733 scope.go:122] "RemoveContainer" containerID="003e624fc8dd9a1606bba949beaaaf1613d680e4f8a551bfe6ca42006fcae53d" Oct 30 20:00:22.787509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.787205 5733 scope.go:122] "RemoveContainer" containerID="df494a82deb23e705fb77e86cdad86b546de86746b1c8b9da8991623520ec5cd" Oct 30 20:00:22.788141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.788048 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:22.808271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.808246 5733 scope.go:122] "RemoveContainer" containerID="f41b8f5bb1aa526ad553e80ed7befe5176cf721c7690ec7dafdd098028b9a7b4" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.915764 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917839 5733 cpu_manager.go:414] "containerMap: removing container" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917863 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917928 5733 cpu_manager.go:414] "containerMap: removing container" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917936 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917956 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917965 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917985 5733 cpu_manager.go:414] "containerMap: removing container" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.917992 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.918256 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.918287 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:22.955974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:22.918299 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8878558a-bcd4-478e-a62e-3edcd26ab88f" containerName="istio-proxy" Oct 30 20:00:23.027735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.026729 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.030533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.030504 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:00:23.036328 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.036301 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:00:23.037995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.037966 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" containerName="cleanup" Oct 30 20:00:23.227018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.044139 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.227018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.198399 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9k9g\" (UniqueName: \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.227018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.198500 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.227018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.204072 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c50e1e\"" Oct 30 20:00:23.227018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.204216 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c50e1e-dockercfg-kk254\"" Oct 30 20:00:23.303476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.301776 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9k9g\" (UniqueName: \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.303476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.301915 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.303476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.301978 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.303476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.302352 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxb4\" (UniqueName: \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.305319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.305264 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.331661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.327829 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:23.331661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.329364 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:00:23.337477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.337412 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9k9g\" (UniqueName: \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\") pod \"payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.398290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.398261 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:23.402771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.402746 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:23.404631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.404599 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxb4\" (UniqueName: \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.405018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.404989 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.407284 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.406106 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.408499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.408474 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:23.421717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.421608 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxb4\" (UniqueName: \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\") pod \"payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.506742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.506712 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\") pod \"d1032716-d9e7-4ff9-8504-760423b63c34\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " Oct 30 20:00:23.506949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.506934 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\" (UniqueName: \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\") pod \"d1032716-d9e7-4ff9-8504-760423b63c34\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " Oct 30 20:00:23.507002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.506988 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\" (UniqueName: \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\") pod \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " Oct 30 20:00:23.507052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.507037 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\") pod \"d1032716-d9e7-4ff9-8504-760423b63c34\" (UID: \"d1032716-d9e7-4ff9-8504-760423b63c34\") " Oct 30 20:00:23.507213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.507198 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\") pod \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " Oct 30 20:00:23.507394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.507378 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\") pod \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\" (UID: \"b5db55fe-d191-45ce-92d6-7b44dc4c68c3\") " Oct 30 20:00:23.508779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.508753 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca" pod "d1032716-d9e7-4ff9-8504-760423b63c34" (UID: "d1032716-d9e7-4ff9-8504-760423b63c34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:23.509433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.509404 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca" pod "b5db55fe-d191-45ce-92d6-7b44dc4c68c3" (UID: "b5db55fe-d191-45ce-92d6-7b44dc4c68c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:23.511678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.511648 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7" pod "d1032716-d9e7-4ff9-8504-760423b63c34" (UID: "d1032716-d9e7-4ff9-8504-760423b63c34"). InnerVolumeSpecName "kube-api-access-2rwk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:23.511951 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.511933 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj" pod "b5db55fe-d191-45ce-92d6-7b44dc4c68c3" (UID: "b5db55fe-d191-45ce-92d6-7b44dc4c68c3"). InnerVolumeSpecName "kube-api-access-zj4tj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:23.515256 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.515236 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca" pod "d1032716-d9e7-4ff9-8504-760423b63c34" (UID: "d1032716-d9e7-4ff9-8504-760423b63c34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:23.516456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.516320 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca" pod "b5db55fe-d191-45ce-92d6-7b44dc4c68c3" (UID: "b5db55fe-d191-45ce-92d6-7b44dc4c68c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:23.545338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.545306 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:23.610963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.610924 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.611206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.610967 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.611206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.610984 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.611206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.611004 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwk7\" (UniqueName: \"kubernetes.io/projected/d1032716-d9e7-4ff9-8504-760423b63c34-kube-api-access-2rwk7\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.611206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.611019 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj4tj\" (UniqueName: \"kubernetes.io/projected/b5db55fe-d191-45ce-92d6-7b44dc4c68c3-kube-api-access-zj4tj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.611206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.611033 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1032716-d9e7-4ff9-8504-760423b63c34-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:23.739685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.739518 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:00:23.740664 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:23.740637 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a663b2_8dd3_4532_9269_a6048f484290.slice/crio-fd88937b9b32543f2a2a9b68d7bbd9d5a39099b905ead385adb5c8e1da04cd56 WatchSource:0}: Error finding container fd88937b9b32543f2a2a9b68d7bbd9d5a39099b905ead385adb5c8e1da04cd56: Status 404 returned error can't find the container with id fd88937b9b32543f2a2a9b68d7bbd9d5a39099b905ead385adb5c8e1da04cd56 Oct 30 20:00:23.780579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.780545 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" Oct 30 20:00:23.780579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.780560 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z" event={"ID":"b5db55fe-d191-45ce-92d6-7b44dc4c68c3","Type":"ContainerDied","Data":"353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9"} Oct 30 20:00:23.780948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.780621 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353c0644f1d5c713be038f24337baae85f117858ec6776a99b75560adafc59a9" Oct 30 20:00:23.783032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.783011 5733 scope.go:122] "RemoveContainer" containerID="8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3" Oct 30 20:00:23.783462 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.783439 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6_odh-ai-gateway-infra(6b553ede-8241-48cb-9d9c-fcb7df9f0a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" Oct 30 20:00:23.785151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.785117 5733 generic.go:356] "Generic (PLEG): container finished" podID="0c972e35-d044-4548-afa9-003279837087" containerID="a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c" exitCode=60 Oct 30 20:00:23.785268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.785203 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerDied","Data":"a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c"} Oct 30 20:00:23.785333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.785281 5733 scope.go:122] "RemoveContainer" containerID="0337acdc42ecf68ae3e9dd4aae7bbe63dfab4a627d87af5792f27031d9540745" Oct 30 20:00:23.786229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.785690 5733 scope.go:122] "RemoveContainer" containerID="a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c" Oct 30 20:00:23.786229 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.786167 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6_odh-ai-gateway-infra(0c972e35-d044-4548-afa9-003279837087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" podUID="0c972e35-d044-4548-afa9-003279837087" Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.789763 5733 scope.go:122] "RemoveContainer" containerID="d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0" Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.790186 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v_odh-ai-gateway-infra(e578b6ca-d719-4dcf-8d46-d59825404e14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.793187 5733 generic.go:356] "Generic (PLEG): container finished" podID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerID="846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc" exitCode=60 Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.793271 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerDied","Data":"846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc"} Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.793620 5733 scope.go:122] "RemoveContainer" containerID="846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc" Oct 30 20:00:23.794106 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.793968 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-b5dzm_odh-ai-gateway-infra(62521f8f-3368-415c-8e47-d0bc6f967c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" Oct 30 20:00:23.795046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.795020 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" event={"ID":"d1032716-d9e7-4ff9-8504-760423b63c34","Type":"ContainerDied","Data":"fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93"} Oct 30 20:00:23.795046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.795028 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx" Oct 30 20:00:23.795247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.795121 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd33ddb0d130458636f5064afc75d87d6b58b016c6ab759a232284000c59e93" Oct 30 20:00:23.803098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.801743 5733 scope.go:122] "RemoveContainer" containerID="89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd" Oct 30 20:00:23.803098 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.802118 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j_odh-ai-gateway-infra(9b5b7900-2d3a-44bc-a3e4-08b47b4c98db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" Oct 30 20:00:23.804305 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.804287 5733 scope.go:122] "RemoveContainer" containerID="8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b" Oct 30 20:00:23.804560 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.804538 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9_odh-ai-gateway-infra(257ce18b-20a1-4cd6-9745-47d82cc229ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" Oct 30 20:00:23.806479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.806454 5733 generic.go:356] "Generic (PLEG): container finished" podID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerID="15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14" exitCode=60 Oct 30 20:00:23.806572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.806534 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerDied","Data":"15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14"} Oct 30 20:00:23.806833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.806810 5733 scope.go:122] "RemoveContainer" containerID="15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14" Oct 30 20:00:23.807162 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:23.807137 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd_odh-ai-gateway-infra(79ebb26b-cecd-46fd-a23d-368a78afdf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" Oct 30 20:00:23.809755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.809729 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" event={"ID":"85a663b2-8dd3-4532-9269-a6048f484290","Type":"ContainerStarted","Data":"fd88937b9b32543f2a2a9b68d7bbd9d5a39099b905ead385adb5c8e1da04cd56"} Oct 30 20:00:23.812206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.812187 5733 scope.go:122] "RemoveContainer" containerID="4f20801c481f01072679e6461d829a6645b123c3a97a78dcb7c53d4600301721" Oct 30 20:00:23.832413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.832386 5733 scope.go:122] "RemoveContainer" containerID="cc75adc28e36a56c4d5614d90e14a892f2bd5f0e909b386c2e476de2834056ed" Oct 30 20:00:23.922602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:23.922570 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:00:24.050522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.050482 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:24.054901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.054870 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c8aff4-864b-4402-add4-b3cbe146bdc5" path="/var/lib/kubelet/pods/73c8aff4-864b-4402-add4-b3cbe146bdc5/volumes" Oct 30 20:00:24.824321 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.824278 5733 scope.go:122] "RemoveContainer" containerID="a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c" Oct 30 20:00:24.824791 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:24.824765 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6_odh-ai-gateway-infra(0c972e35-d044-4548-afa9-003279837087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" podUID="0c972e35-d044-4548-afa9-003279837087" Oct 30 20:00:24.826172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.826152 5733 scope.go:122] "RemoveContainer" containerID="846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc" Oct 30 20:00:24.826564 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:24.826539 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-b5dzm_odh-ai-gateway-infra(62521f8f-3368-415c-8e47-d0bc6f967c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" Oct 30 20:00:24.828109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.828049 5733 scope.go:122] "RemoveContainer" containerID="15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14" Oct 30 20:00:24.828478 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:24.828451 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd_odh-ai-gateway-infra(79ebb26b-cecd-46fd-a23d-368a78afdf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" Oct 30 20:00:24.829762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.829735 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" event={"ID":"b0c0ab13-e06c-433e-995c-ba78139f9da1","Type":"ContainerStarted","Data":"ec7ba420ef29488248eeea9f351a7c0249cd866b9c8d71d71a535f547e25d95c"} Oct 30 20:00:24.829858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.829830 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:24.829901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.829860 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" event={"ID":"b0c0ab13-e06c-433e-995c-ba78139f9da1","Type":"ContainerStarted","Data":"1aa24a4a3ec62e0d388e535ae71b20428a0383e247a8b15c7cee9e18859a5a3a"} Oct 30 20:00:24.831607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.831588 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" event={"ID":"85a663b2-8dd3-4532-9269-a6048f484290","Type":"ContainerStarted","Data":"2ac925cb55703aab8e717fa2e268134868cd38fded682df87e7288cb7597ad91"} Oct 30 20:00:24.831714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.831694 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:24.931158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.931100 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" podStartSLOduration=2.931083392 podStartE2EDuration="2.931083392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:24.928016841 +0000 UTC m=+1784.995852410" watchObservedRunningTime="2026-10-30 20:00:24.931083392 +0000 UTC m=+1784.998918957" Oct 30 20:00:24.967810 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:24.967757 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" podStartSLOduration=2.967740382 podStartE2EDuration="2.967740382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:24.962437606 +0000 UTC m=+1785.030273175" watchObservedRunningTime="2026-10-30 20:00:24.967740382 +0000 UTC m=+1785.035575949" Oct 30 20:00:25.291183 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:25.291145 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.389267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:26.389235 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 20:00:26.402504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:26.402467 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 20:00:26.517584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:26.517547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.691255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:26.691172 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:27.881614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.881568 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:00:27.883641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.883614 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b5db55fe-d191-45ce-92d6-7b44dc4c68c3" containerName="revoke-keys" Oct 30 20:00:27.883811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.883795 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5db55fe-d191-45ce-92d6-7b44dc4c68c3" containerName="revoke-keys" Oct 30 20:00:27.883943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.883930 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d1032716-d9e7-4ff9-8504-760423b63c34" containerName="revoke-keys" Oct 30 20:00:27.884034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.884021 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1032716-d9e7-4ff9-8504-760423b63c34" containerName="revoke-keys" Oct 30 20:00:27.884356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.884337 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5db55fe-d191-45ce-92d6-7b44dc4c68c3" containerName="revoke-keys" Oct 30 20:00:27.884471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.884459 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1032716-d9e7-4ff9-8504-760423b63c34" containerName="revoke-keys" Oct 30 20:00:27.896159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.896129 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:00:27.896290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.896247 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:27.905508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.905474 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae43d70e-serving-cert\"" Oct 30 20:00:27.915716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.915518 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:27.916010 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.915983 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87kzz\" (UniqueName: \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:27.916186 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.916067 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:27.916253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:27.916201 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.017696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.017660 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.017885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.017776 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.017885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.017856 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.018052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.018039 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87kzz\" (UniqueName: \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.018549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.018523 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.020350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.020323 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.020555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.020532 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.034833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.034805 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87kzz\" (UniqueName: \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\") pod \"maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.073089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.068802 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.219264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.219183 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:28.237717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.237682 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.539711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.539670 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:00:28.539892 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:28.539867 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a11fe9c_b2a8_4a85_88f7_d766e1d0e824.slice/crio-58c2a853981c8461c5e0bf4ef493283cc617b31bf3c28ca0942927670edeaa15 WatchSource:0}: Error finding container 58c2a853981c8461c5e0bf4ef493283cc617b31bf3c28ca0942927670edeaa15: Status 404 returned error can't find the container with id 58c2a853981c8461c5e0bf4ef493283cc617b31bf3c28ca0942927670edeaa15 Oct 30 20:00:28.930594 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:28.930555 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" event={"ID":"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824","Type":"ContainerStarted","Data":"58c2a853981c8461c5e0bf4ef493283cc617b31bf3c28ca0942927670edeaa15"} Oct 30 20:00:29.422468 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:29.422428 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:00:29.596987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:29.596948 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.747168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:29.747083 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.944650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:29.944600 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" event={"ID":"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824","Type":"ContainerStarted","Data":"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d"} Oct 30 20:00:29.945131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:29.944682 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:29.999083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:29.998973 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" podStartSLOduration=2.503540638 podStartE2EDuration="2.998960051s" totalImagesPullingTime="495.419413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:29.994978379 +0000 UTC m=+1790.062813976" watchObservedRunningTime="2026-10-30 20:00:29.998960051 +0000 UTC m=+1790.066795647" Oct 30 20:00:30.445104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:30.445073 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:00:31.205994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.205963 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.389947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.389891 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" podUID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" containerName="payload-processing" containerID="cri-o://39ffabc117d6fb2672dd3348cd1e391e49e5d68b7c55cd3fca32be370e2d0754" gracePeriod=25 Oct 30 20:00:31.403149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.403109 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" podUID="6844dca4-0d51-4f57-911f-7238e5deebc6" containerName="payload-pre-processing" containerID="cri-o://7ae9ab0841d0b01299c4283fb3ef23717984d92af0e04ce55ef204f3f8a0ebc7" gracePeriod=25 Oct 30 20:00:31.829177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.827401 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:00:31.831141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.829573 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:00:31.860699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.860659 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:00:31.980010 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.979930 5733 generic.go:356] "Generic (PLEG): container finished" podID="6844dca4-0d51-4f57-911f-7238e5deebc6" containerID="7ae9ab0841d0b01299c4283fb3ef23717984d92af0e04ce55ef204f3f8a0ebc7" exitCode=0 Oct 30 20:00:31.980220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.980103 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" event={"ID":"6844dca4-0d51-4f57-911f-7238e5deebc6","Type":"ContainerDied","Data":"7ae9ab0841d0b01299c4283fb3ef23717984d92af0e04ce55ef204f3f8a0ebc7"} Oct 30 20:00:31.984896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.984183 5733 generic.go:356] "Generic (PLEG): container finished" podID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" containerID="39ffabc117d6fb2672dd3348cd1e391e49e5d68b7c55cd3fca32be370e2d0754" exitCode=0 Oct 30 20:00:31.984896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:31.984504 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" event={"ID":"e4d4bbb7-e918-421a-859d-0e1c072ee7fd","Type":"ContainerDied","Data":"39ffabc117d6fb2672dd3348cd1e391e49e5d68b7c55cd3fca32be370e2d0754"} Oct 30 20:00:32.172240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.172212 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 20:00:32.250684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.250633 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\") pod \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " Oct 30 20:00:32.251031 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.250764 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\" (UniqueName: \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\") pod \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\" (UID: \"e4d4bbb7-e918-421a-859d-0e1c072ee7fd\") " Oct 30 20:00:32.253903 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.253876 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk" pod "e4d4bbb7-e918-421a-859d-0e1c072ee7fd" (UID: "e4d4bbb7-e918-421a-859d-0e1c072ee7fd"). InnerVolumeSpecName "kube-api-access-vvgsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.254238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.254216 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume" pod "e4d4bbb7-e918-421a-859d-0e1c072ee7fd" (UID: "e4d4bbb7-e918-421a-859d-0e1c072ee7fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:32.258750 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.258730 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 20:00:32.352359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.352321 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\" (UniqueName: \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\") pod \"6844dca4-0d51-4f57-911f-7238e5deebc6\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " Oct 30 20:00:32.352561 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.352508 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\") pod \"6844dca4-0d51-4f57-911f-7238e5deebc6\" (UID: \"6844dca4-0d51-4f57-911f-7238e5deebc6\") " Oct 30 20:00:32.352917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.352888 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume" pod "6844dca4-0d51-4f57-911f-7238e5deebc6" (UID: "6844dca4-0d51-4f57-911f-7238e5deebc6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:32.353603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.353573 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgsk\" (UniqueName: \"kubernetes.io/projected/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-kube-api-access-vvgsk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:32.353736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.353637 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6844dca4-0d51-4f57-911f-7238e5deebc6-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:32.353736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.353655 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d4bbb7-e918-421a-859d-0e1c072ee7fd-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:32.356426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.356391 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx" pod "6844dca4-0d51-4f57-911f-7238e5deebc6" (UID: "6844dca4-0d51-4f57-911f-7238e5deebc6"). InnerVolumeSpecName "kube-api-access-s6hxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:32.365588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.365525 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.455716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.455688 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6hxx\" (UniqueName: \"kubernetes.io/projected/6844dca4-0d51-4f57-911f-7238e5deebc6-kube-api-access-s6hxx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:32.548628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.547893 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.996217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.996185 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" Oct 30 20:00:32.996217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.996209 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd" event={"ID":"6844dca4-0d51-4f57-911f-7238e5deebc6","Type":"ContainerDied","Data":"ccf6b8be945c54348c90b124b4f2bd43fcb301c7b53db9ea07bc19d2baae1e5f"} Oct 30 20:00:32.996485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.996279 5733 scope.go:122] "RemoveContainer" containerID="7ae9ab0841d0b01299c4283fb3ef23717984d92af0e04ce55ef204f3f8a0ebc7" Oct 30 20:00:32.998398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.998354 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" event={"ID":"e4d4bbb7-e918-421a-859d-0e1c072ee7fd","Type":"ContainerDied","Data":"6aa7e32af4af7505c80b3992a1196f17a56c947ba5c9f45da828565490cdaa0a"} Oct 30 20:00:32.998514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:32.998364 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4" Oct 30 20:00:33.010379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.010354 5733 scope.go:122] "RemoveContainer" containerID="39ffabc117d6fb2672dd3348cd1e391e49e5d68b7c55cd3fca32be370e2d0754" Oct 30 20:00:33.032526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.032493 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 20:00:33.041685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.041657 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-bdd3b5-788c79fc67-42prd"] Oct 30 20:00:33.054747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.054687 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 20:00:33.059884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.059853 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-bdd3b5-5fd5548d5d-fllv4"] Oct 30 20:00:33.710958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:33.709702 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:34.043191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.043100 5733 scope.go:122] "RemoveContainer" containerID="d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0" Oct 30 20:00:34.043687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.043660 5733 scope.go:122] "RemoveContainer" containerID="846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc" Oct 30 20:00:34.043929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.043904 5733 scope.go:122] "RemoveContainer" containerID="8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b" Oct 30 20:00:34.044748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.044719 5733 scope.go:122] "RemoveContainer" containerID="89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd" Oct 30 20:00:34.058587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.058541 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6844dca4-0d51-4f57-911f-7238e5deebc6" path="/var/lib/kubelet/pods/6844dca4-0d51-4f57-911f-7238e5deebc6/volumes" Oct 30 20:00:34.059289 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.059267 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" path="/var/lib/kubelet/pods/e4d4bbb7-e918-421a-859d-0e1c072ee7fd/volumes" Oct 30 20:00:34.688050 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:34.688000 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.030926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.030836 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerStarted","Data":"791d202f4dc714db1ab62039e951887c9e6100a6f37c80e568eed2bb6ed3a052"} Oct 30 20:00:35.033658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.033341 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerStarted","Data":"9527680463bdc2ced842dea2fcdb0846032ce4e4abd0d6efe9b1f4bcf019150e"} Oct 30 20:00:35.035569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.035526 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerStarted","Data":"54b43681f3499a8e67a19aff42b5ba6df48d97cdd47bb52706067b9768715177"} Oct 30 20:00:35.038421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.038397 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerStarted","Data":"14cbb3622b09c9db49a868399c38e18c9b71bc7013437f2f9d6ba4a0a3b01ab0"} Oct 30 20:00:35.547871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.546377 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.845361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.845323 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:00:35.845562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.845417 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:00:35.963227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:35.963193 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:00:36.049043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.047900 5733 scope.go:122] "RemoveContainer" containerID="8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3" Oct 30 20:00:36.071216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.068879 5733 generic.go:356] "Generic (PLEG): container finished" podID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerID="791d202f4dc714db1ab62039e951887c9e6100a6f37c80e568eed2bb6ed3a052" exitCode=60 Oct 30 20:00:36.071216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.068950 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerDied","Data":"791d202f4dc714db1ab62039e951887c9e6100a6f37c80e568eed2bb6ed3a052"} Oct 30 20:00:36.071216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.069126 5733 scope.go:122] "RemoveContainer" containerID="d3416503f13cea71d1d7dfb318f985cd7983c462f4e55945a09ec3af5c0f3ac0" Oct 30 20:00:36.071216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.069538 5733 scope.go:122] "RemoveContainer" containerID="791d202f4dc714db1ab62039e951887c9e6100a6f37c80e568eed2bb6ed3a052" Oct 30 20:00:36.071216 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:36.070563 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v_odh-ai-gateway-infra(e578b6ca-d719-4dcf-8d46-d59825404e14)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" Oct 30 20:00:36.073667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.073641 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:36.080883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.075324 5733 generic.go:356] "Generic (PLEG): container finished" podID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerID="9527680463bdc2ced842dea2fcdb0846032ce4e4abd0d6efe9b1f4bcf019150e" exitCode=60 Oct 30 20:00:36.080883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.075429 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerDied","Data":"9527680463bdc2ced842dea2fcdb0846032ce4e4abd0d6efe9b1f4bcf019150e"} Oct 30 20:00:36.080883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.075910 5733 scope.go:122] "RemoveContainer" containerID="9527680463bdc2ced842dea2fcdb0846032ce4e4abd0d6efe9b1f4bcf019150e" Oct 30 20:00:36.080883 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:36.076586 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-b5dzm_odh-ai-gateway-infra(62521f8f-3368-415c-8e47-d0bc6f967c5d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" Oct 30 20:00:36.081890 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.081440 5733 generic.go:356] "Generic (PLEG): container finished" podID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerID="54b43681f3499a8e67a19aff42b5ba6df48d97cdd47bb52706067b9768715177" exitCode=60 Oct 30 20:00:36.081890 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.081560 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerDied","Data":"54b43681f3499a8e67a19aff42b5ba6df48d97cdd47bb52706067b9768715177"} Oct 30 20:00:36.082039 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.082020 5733 scope.go:122] "RemoveContainer" containerID="54b43681f3499a8e67a19aff42b5ba6df48d97cdd47bb52706067b9768715177" Oct 30 20:00:36.082414 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:36.082391 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j_odh-ai-gateway-infra(9b5b7900-2d3a-44bc-a3e4-08b47b4c98db)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" Oct 30 20:00:36.085943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.085903 5733 generic.go:356] "Generic (PLEG): container finished" podID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerID="14cbb3622b09c9db49a868399c38e18c9b71bc7013437f2f9d6ba4a0a3b01ab0" exitCode=60 Oct 30 20:00:36.086311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.086290 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerDied","Data":"14cbb3622b09c9db49a868399c38e18c9b71bc7013437f2f9d6ba4a0a3b01ab0"} Oct 30 20:00:36.086907 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.086871 5733 scope.go:122] "RemoveContainer" containerID="14cbb3622b09c9db49a868399c38e18c9b71bc7013437f2f9d6ba4a0a3b01ab0" Oct 30 20:00:36.088585 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:36.088080 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9_odh-ai-gateway-infra(257ce18b-20a1-4cd6-9745-47d82cc229ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" Oct 30 20:00:36.092229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.092177 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:36.139465 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.139439 5733 scope.go:122] "RemoveContainer" containerID="846d19416984a5f9c337fc28f3fad871a82493e838b8efe7a68ff1a69a31c1dc" Oct 30 20:00:36.150766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.150737 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:36.162947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.162917 5733 scope.go:122] "RemoveContainer" containerID="89d73b24328708cf3245ec03a0118ff8d5b58c4a0f2e5d1d72670774f92a39dd" Oct 30 20:00:36.168635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.168607 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:36.179369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.179348 5733 scope.go:122] "RemoveContainer" containerID="8b6cebba7c3ef9d244d83f5c2f5840730f402bc95ee03cdab076fe86d7b9b17b" Oct 30 20:00:36.416200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:36.414485 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:37.042870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.042825 5733 scope.go:122] "RemoveContainer" containerID="a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c" Oct 30 20:00:37.043359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.043334 5733 scope.go:122] "RemoveContainer" containerID="15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14" Oct 30 20:00:37.122852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.122816 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerStarted","Data":"fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216"} Oct 30 20:00:37.331871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.331806 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:37.471827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.471406 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:37.561817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.561705 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:37.590579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.590549 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\" (UniqueName: \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\") pod \"62521f8f-3368-415c-8e47-d0bc6f967c5d\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " Oct 30 20:00:37.591555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.591531 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\") pod \"62521f8f-3368-415c-8e47-d0bc6f967c5d\" (UID: \"62521f8f-3368-415c-8e47-d0bc6f967c5d\") " Oct 30 20:00:37.592912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.592890 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca" pod "62521f8f-3368-415c-8e47-d0bc6f967c5d" (UID: "62521f8f-3368-415c-8e47-d0bc6f967c5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:37.593697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.593677 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp" pod "62521f8f-3368-415c-8e47-d0bc6f967c5d" (UID: "62521f8f-3368-415c-8e47-d0bc6f967c5d"). InnerVolumeSpecName "kube-api-access-svtdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:37.693479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.693440 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\") pod \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " Oct 30 20:00:37.693706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.693686 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\" (UniqueName: \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\") pod \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\" (UID: \"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db\") " Oct 30 20:00:37.693887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.693859 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca" pod "9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" (UID: "9b5b7900-2d3a-44bc-a3e4-08b47b4c98db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:37.694596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.694571 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62521f8f-3368-415c-8e47-d0bc6f967c5d-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.694717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.694598 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.694717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.694612 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtdp\" (UniqueName: \"kubernetes.io/projected/62521f8f-3368-415c-8e47-d0bc6f967c5d-kube-api-access-svtdp\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.697558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.697526 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm" pod "9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" (UID: "9b5b7900-2d3a-44bc-a3e4-08b47b4c98db"). InnerVolumeSpecName "kube-api-access-ws4wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:37.706762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.706733 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:37.764411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.762042 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:37.796131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.796098 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\" (UniqueName: \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\") pod \"e578b6ca-d719-4dcf-8d46-d59825404e14\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " Oct 30 20:00:37.796290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.796172 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\") pod \"e578b6ca-d719-4dcf-8d46-d59825404e14\" (UID: \"e578b6ca-d719-4dcf-8d46-d59825404e14\") " Oct 30 20:00:37.796522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.796499 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca" pod "e578b6ca-d719-4dcf-8d46-d59825404e14" (UID: "e578b6ca-d719-4dcf-8d46-d59825404e14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:37.797257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.797230 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws4wm\" (UniqueName: \"kubernetes.io/projected/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db-kube-api-access-ws4wm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.797257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.797256 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e578b6ca-d719-4dcf-8d46-d59825404e14-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.798755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.798720 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs" pod "e578b6ca-d719-4dcf-8d46-d59825404e14" (UID: "e578b6ca-d719-4dcf-8d46-d59825404e14"). InnerVolumeSpecName "kube-api-access-gsmfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:37.898379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.898343 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\" (UniqueName: \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\") pod \"257ce18b-20a1-4cd6-9745-47d82cc229ea\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " Oct 30 20:00:37.898720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.898703 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\") pod \"257ce18b-20a1-4cd6-9745-47d82cc229ea\" (UID: \"257ce18b-20a1-4cd6-9745-47d82cc229ea\") " Oct 30 20:00:37.899093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.899066 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca" pod "257ce18b-20a1-4cd6-9745-47d82cc229ea" (UID: "257ce18b-20a1-4cd6-9745-47d82cc229ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:37.899448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.899433 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsmfs\" (UniqueName: \"kubernetes.io/projected/e578b6ca-d719-4dcf-8d46-d59825404e14-kube-api-access-gsmfs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.899487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.899456 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/257ce18b-20a1-4cd6-9745-47d82cc229ea-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:37.900408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:37.900391 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv" pod "257ce18b-20a1-4cd6-9745-47d82cc229ea" (UID: "257ce18b-20a1-4cd6-9745-47d82cc229ea"). InnerVolumeSpecName "kube-api-access-hm6rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:38.001245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.001209 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm6rv\" (UniqueName: \"kubernetes.io/projected/257ce18b-20a1-4cd6-9745-47d82cc229ea-kube-api-access-hm6rv\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:38.140865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.140824 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" Oct 30 20:00:38.140865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.140862 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j" event={"ID":"9b5b7900-2d3a-44bc-a3e4-08b47b4c98db","Type":"ContainerDied","Data":"a8030dbfc81446789d9c95fe58890b42697fb5eb50cf50393f0581ebc9081ebd"} Oct 30 20:00:38.141405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.140939 5733 scope.go:122] "RemoveContainer" containerID="54b43681f3499a8e67a19aff42b5ba6df48d97cdd47bb52706067b9768715177" Oct 30 20:00:38.142884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.142797 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" Oct 30 20:00:38.143482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.142820 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9" event={"ID":"257ce18b-20a1-4cd6-9745-47d82cc229ea","Type":"ContainerDied","Data":"f976a97a57f9ec716ae90bbe5a8edf43be861921d7c86f15a4192498ed36a4f2"} Oct 30 20:00:38.146257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.146232 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerStarted","Data":"9e8cb6530de434e252890f72cfbb0ad9b54d5b3689dfcc653f71dcd6ed890c8f"} Oct 30 20:00:38.148994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.148935 5733 generic.go:356] "Generic (PLEG): container finished" podID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerID="fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216" exitCode=60 Oct 30 20:00:38.149129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.149002 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerDied","Data":"fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216"} Oct 30 20:00:38.149504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.149481 5733 scope.go:122] "RemoveContainer" containerID="fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216" Oct 30 20:00:38.149830 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:38.149803 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6_odh-ai-gateway-infra(6b553ede-8241-48cb-9d9c-fcb7df9f0a90)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" Oct 30 20:00:38.151621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.151601 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerStarted","Data":"65a4aea177943a9476caeade27daa84749a3e74c5349e694d4777e94b2cb9b27"} Oct 30 20:00:38.153319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.153286 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" Oct 30 20:00:38.153426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.153337 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v" event={"ID":"e578b6ca-d719-4dcf-8d46-d59825404e14","Type":"ContainerDied","Data":"0ecc27e7167b2f166273d0762d6789147f12bb73e15d01148a6499ab8a1bb8e9"} Oct 30 20:00:38.155739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.155715 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" event={"ID":"62521f8f-3368-415c-8e47-d0bc6f967c5d","Type":"ContainerDied","Data":"95fa492e9a05ffd4316f08fc73ce414d24da8ca401c1763b19dc479f87172ac8"} Oct 30 20:00:38.155839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.155742 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm" Oct 30 20:00:38.155967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.155923 5733 scope.go:122] "RemoveContainer" containerID="14cbb3622b09c9db49a868399c38e18c9b71bc7013437f2f9d6ba4a0a3b01ab0" Oct 30 20:00:38.167569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.167549 5733 scope.go:122] "RemoveContainer" containerID="8bd483bcaea61aeecf37bdaaec99620297b146c06286a87dc21d6d933f4f51c3" Oct 30 20:00:38.183862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.183838 5733 scope.go:122] "RemoveContainer" containerID="791d202f4dc714db1ab62039e951887c9e6100a6f37c80e568eed2bb6ed3a052" Oct 30 20:00:38.195337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.195320 5733 scope.go:122] "RemoveContainer" containerID="9527680463bdc2ced842dea2fcdb0846032ce4e4abd0d6efe9b1f4bcf019150e" Oct 30 20:00:38.225811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.225434 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:38.231985 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.231957 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f5d176-29889840-4n96j"] Oct 30 20:00:38.287500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.287471 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:38.332233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.332196 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:38.340373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.340344 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bdd3b5-29889840-x2pr9"] Oct 30 20:00:38.350549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.350520 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:38.356007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.355980 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae2147-29889840-dz96v"] Oct 30 20:00:38.371096 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.371038 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:38.375885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.375862 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-b5dzm"] Oct 30 20:00:38.533753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:38.533625 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.185568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.185532 5733 generic.go:356] "Generic (PLEG): container finished" podID="0c972e35-d044-4548-afa9-003279837087" containerID="65a4aea177943a9476caeade27daa84749a3e74c5349e694d4777e94b2cb9b27" exitCode=60 Oct 30 20:00:39.185924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.185608 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerDied","Data":"65a4aea177943a9476caeade27daa84749a3e74c5349e694d4777e94b2cb9b27"} Oct 30 20:00:39.185924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.185746 5733 scope.go:122] "RemoveContainer" containerID="a49400f31a8c2c3504bf7af5c51dcdf2905e1e585d9869ad74ad7c6f9501b18c" Oct 30 20:00:39.186216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.186198 5733 scope.go:122] "RemoveContainer" containerID="65a4aea177943a9476caeade27daa84749a3e74c5349e694d4777e94b2cb9b27" Oct 30 20:00:39.186703 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:39.186679 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6_odh-ai-gateway-infra(0c972e35-d044-4548-afa9-003279837087)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" podUID="0c972e35-d044-4548-afa9-003279837087" Oct 30 20:00:39.192600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.192572 5733 generic.go:356] "Generic (PLEG): container finished" podID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerID="9e8cb6530de434e252890f72cfbb0ad9b54d5b3689dfcc653f71dcd6ed890c8f" exitCode=60 Oct 30 20:00:39.192724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.192648 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerDied","Data":"9e8cb6530de434e252890f72cfbb0ad9b54d5b3689dfcc653f71dcd6ed890c8f"} Oct 30 20:00:39.193235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.193213 5733 scope.go:122] "RemoveContainer" containerID="9e8cb6530de434e252890f72cfbb0ad9b54d5b3689dfcc653f71dcd6ed890c8f" Oct 30 20:00:39.193686 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:39.193664 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd_odh-ai-gateway-infra(79ebb26b-cecd-46fd-a23d-368a78afdf03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" Oct 30 20:00:39.201971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.201942 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:39.202670 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.202651 5733 scope.go:122] "RemoveContainer" containerID="15d2b2b431e8e65c7e93aadc4899da7fa7770304c108930386d0eec28496cc14" Oct 30 20:00:39.350780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.350701 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:39.419094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.419021 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:39.455146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.454850 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\" (UniqueName: \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\") pod \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " Oct 30 20:00:39.455323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.455159 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\") pod \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\" (UID: \"6b553ede-8241-48cb-9d9c-fcb7df9f0a90\") " Oct 30 20:00:39.456146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.456117 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca" pod "6b553ede-8241-48cb-9d9c-fcb7df9f0a90" (UID: "6b553ede-8241-48cb-9d9c-fcb7df9f0a90"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:39.460961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.460931 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx" pod "6b553ede-8241-48cb-9d9c-fcb7df9f0a90" (UID: "6b553ede-8241-48cb-9d9c-fcb7df9f0a90"). InnerVolumeSpecName "kube-api-access-7fvgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:39.462638 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.462609 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.569875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.569826 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:39.569875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:39.569866 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fvgx\" (UniqueName: \"kubernetes.io/projected/6b553ede-8241-48cb-9d9c-fcb7df9f0a90-kube-api-access-7fvgx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:40.059424 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.059384 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" path="/var/lib/kubelet/pods/257ce18b-20a1-4cd6-9745-47d82cc229ea/volumes" Oct 30 20:00:40.060047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.060021 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" path="/var/lib/kubelet/pods/62521f8f-3368-415c-8e47-d0bc6f967c5d/volumes" Oct 30 20:00:40.060687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.060664 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" path="/var/lib/kubelet/pods/9b5b7900-2d3a-44bc-a3e4-08b47b4c98db/volumes" Oct 30 20:00:40.061475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.061450 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" path="/var/lib/kubelet/pods/e578b6ca-d719-4dcf-8d46-d59825404e14/volumes" Oct 30 20:00:40.367137 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.367106 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" event={"ID":"6b553ede-8241-48cb-9d9c-fcb7df9f0a90","Type":"ContainerDied","Data":"f58a558dcb263242f65069bf3de993cfdc813fb39660b4e501811da84601891a"} Oct 30 20:00:40.367573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.367554 5733 scope.go:122] "RemoveContainer" containerID="fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216" Oct 30 20:00:40.367767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.367754 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6" Oct 30 20:00:40.536117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.526572 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:40.536117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.533355 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-eaa5cd-29889840-nlwv6"] Oct 30 20:00:40.628428 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:40.627611 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216\": container with ID starting with fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216 not found: ID does not exist" containerID="fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216" Oct 30 20:00:40.628428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.627660 5733 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216\": container with ID starting with fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216 not found: ID does not exist" Oct 30 20:00:40.628428 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:40.627704 5733 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216\": container with ID starting with fb3631e08d732e320bb4018d68ee11130da8364030a90f87c95b1d5a59151216 not found: ID does not exist" Oct 30 20:00:40.749097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.741997 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.871153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.870374 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:40.872969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.872937 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:40.928272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.927622 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\" (UniqueName: \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\") pod \"79ebb26b-cecd-46fd-a23d-368a78afdf03\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " Oct 30 20:00:40.928272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.927863 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\") pod \"0c972e35-d044-4548-afa9-003279837087\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " Oct 30 20:00:40.928272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.928155 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\") pod \"79ebb26b-cecd-46fd-a23d-368a78afdf03\" (UID: \"79ebb26b-cecd-46fd-a23d-368a78afdf03\") " Oct 30 20:00:40.928272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.928228 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\" (UniqueName: \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\") pod \"0c972e35-d044-4548-afa9-003279837087\" (UID: \"0c972e35-d044-4548-afa9-003279837087\") " Oct 30 20:00:40.932538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.932154 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca" pod "0c972e35-d044-4548-afa9-003279837087" (UID: "0c972e35-d044-4548-afa9-003279837087"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:40.937440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.932829 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca" pod "79ebb26b-cecd-46fd-a23d-368a78afdf03" (UID: "79ebb26b-cecd-46fd-a23d-368a78afdf03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:40.937440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.933039 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2" pod "0c972e35-d044-4548-afa9-003279837087" (UID: "0c972e35-d044-4548-afa9-003279837087"). InnerVolumeSpecName "kube-api-access-zxrx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:40.958848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:40.958798 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh" pod "79ebb26b-cecd-46fd-a23d-368a78afdf03" (UID: "79ebb26b-cecd-46fd-a23d-368a78afdf03"). InnerVolumeSpecName "kube-api-access-ptqbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:41.031510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.031475 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79ebb26b-cecd-46fd-a23d-368a78afdf03-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:41.031510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.031510 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxrx2\" (UniqueName: \"kubernetes.io/projected/0c972e35-d044-4548-afa9-003279837087-kube-api-access-zxrx2\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:41.031766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.031527 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqbh\" (UniqueName: \"kubernetes.io/projected/79ebb26b-cecd-46fd-a23d-368a78afdf03-kube-api-access-ptqbh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:41.031766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.031541 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c972e35-d044-4548-afa9-003279837087-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:41.432082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.422899 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" event={"ID":"79ebb26b-cecd-46fd-a23d-368a78afdf03","Type":"ContainerDied","Data":"165deadddff5825089f0f9a332354aadb1669b0d9e8fb2aff7e0919a37732fbd"} Oct 30 20:00:41.432082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.423445 5733 scope.go:122] "RemoveContainer" containerID="9e8cb6530de434e252890f72cfbb0ad9b54d5b3689dfcc653f71dcd6ed890c8f" Oct 30 20:00:41.432082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.422922 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd" Oct 30 20:00:41.494108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.486611 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" event={"ID":"0c972e35-d044-4548-afa9-003279837087","Type":"ContainerDied","Data":"f51cbcfa52202c03c8ca7ae665f1df6281e5d29c4324fbb4493c6ee67b59e329"} Oct 30 20:00:41.494108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.486746 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6" Oct 30 20:00:41.625304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.625245 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:41.649386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.647002 5733 scope.go:122] "RemoveContainer" containerID="65a4aea177943a9476caeade27daa84749a3e74c5349e694d4777e94b2cb9b27" Oct 30 20:00:41.704025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.703983 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3f57c5-29889840-f87dd"] Oct 30 20:00:41.984553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:41.984506 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:42.026304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:42.026257 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3d0a78-29889840-xrpk6"] Oct 30 20:00:42.055776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:42.055737 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c972e35-d044-4548-afa9-003279837087" path="/var/lib/kubelet/pods/0c972e35-d044-4548-afa9-003279837087/volumes" Oct 30 20:00:42.056668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:42.056629 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" path="/var/lib/kubelet/pods/6b553ede-8241-48cb-9d9c-fcb7df9f0a90/volumes" Oct 30 20:00:42.057353 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:42.057326 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" path="/var/lib/kubelet/pods/79ebb26b-cecd-46fd-a23d-368a78afdf03/volumes" Oct 30 20:00:42.433799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:42.433716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.303988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:43.303895 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.323658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:43.323625 5733 scope.go:122] "RemoveContainer" containerID="308687b11b045687e7b67987504d93e9627da35f9e918779816fc4ef541a2ea7" Oct 30 20:00:43.341420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:43.341389 5733 scope.go:122] "RemoveContainer" containerID="1e009ef97ae8f3ad95e4903f5777a1e58cc5d9d8c27ff6115b3404517f0891de" Oct 30 20:00:43.383155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:43.381719 5733 scope.go:122] "RemoveContainer" containerID="1e89fea7b5388508d0c4711700312ba047f8b15c153b497725162a8f4ee64694" Oct 30 20:00:43.396708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:43.396681 5733 scope.go:122] "RemoveContainer" containerID="fe1945927c90b8c4e97387e7edaa317f6d2ce3e1f992d01a8f4f8d3cea35929c" Oct 30 20:00:44.466030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:44.465991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.607594 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:44.607557 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.036425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:46.036390 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.903396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:46.903359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.144719 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:48.144680 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.009212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:49.008537 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:50.164340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:50.164303 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.022143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:51.016709 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.859081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:51.857745 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:52.218210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:52.209578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.350598 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.351944 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6844dca4-0d51-4f57-911f-7238e5deebc6" containerName="payload-pre-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.351966 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6844dca4-0d51-4f57-911f-7238e5deebc6" containerName="payload-pre-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.351986 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" containerName="payload-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.351995 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" containerName="payload-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352014 5733 cpu_manager.go:414] "containerMap: removing container" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352024 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352039 5733 cpu_manager.go:414] "containerMap: removing container" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352047 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352080 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352088 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352103 5733 cpu_manager.go:414] "containerMap: removing container" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352110 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352124 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352132 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352149 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352156 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352170 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352177 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352192 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352200 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352211 5733 cpu_manager.go:414] "containerMap: removing container" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352219 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352253 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352262 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352296 5733 cpu_manager.go:414] "containerMap: removing container" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352304 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352321 5733 cpu_manager.go:414] "containerMap: removing container" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352328 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352339 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352345 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352361 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352367 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352380 5733 cpu_manager.go:414] "containerMap: removing container" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352386 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352600 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352617 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352627 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352642 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352650 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352660 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352668 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352679 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352690 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352700 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352712 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352722 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6844dca4-0d51-4f57-911f-7238e5deebc6" containerName="payload-pre-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352735 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d4bbb7-e918-421a-859d-0e1c072ee7fd" containerName="payload-processing" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352748 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352757 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352767 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352775 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352784 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:00:53.355392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.352797 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:00:53.382148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.382115 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:00:53.382289 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.382221 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.457293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.457250 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dd1b04-openshift-default-dockercfg-dhzsm\"" Oct 30 20:00:53.493195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493157 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493206 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493314 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493443 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493417 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493481 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493446 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493569 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493679 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r9fl\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493739 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.493870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.493856 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.518990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.518954 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.596445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596295 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596352 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596457 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596556 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r9fl\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596623 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596732 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.596848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596841 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596881 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.596939 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.597190 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.597444 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.597696 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.597999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.597972 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.598138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.598018 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.602140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.600715 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.602140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.602009 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.613141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.611988 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r9fl\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.613141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.612027 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\") pod \"e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:53.698534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:53.698497 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:54.006384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.006351 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:00:54.007492 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:00:54.007459 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd82080_4fc3_44b6_8554_f0548d18605c.slice/crio-b411d2a4e9eda17754e5f4cc4c20378079410e8850a9689a58fa8b9ba9d0e332 WatchSource:0}: Error finding container b411d2a4e9eda17754e5f4cc4c20378079410e8850a9689a58fa8b9ba9d0e332: Status 404 returned error can't find the container with id b411d2a4e9eda17754e5f4cc4c20378079410e8850a9689a58fa8b9ba9d0e332 Oct 30 20:00:54.034352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.012179 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.034352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.012266 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.034352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.012508 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:00:54.493078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.488737 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.749504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.749400 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" event={"ID":"8dd82080-4fc3-44b6-8554-f0548d18605c","Type":"ContainerStarted","Data":"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53"} Oct 30 20:00:54.749504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.749491 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" event={"ID":"8dd82080-4fc3-44b6-8554-f0548d18605c","Type":"ContainerStarted","Data":"b411d2a4e9eda17754e5f4cc4c20378079410e8850a9689a58fa8b9ba9d0e332"} Oct 30 20:00:54.791788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:54.791722 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podStartSLOduration=1.7917032960000001 podStartE2EDuration="1.791703296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:54.790295773 +0000 UTC m=+1814.858131342" watchObservedRunningTime="2026-10-30 20:00:54.791703296 +0000 UTC m=+1814.859538865" Oct 30 20:00:55.267767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.267518 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.466084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.457468 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.653807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.653775 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.702661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.702632 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:00:55.703029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.702993 5733 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:00:55.703255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:55.703215 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:00:56.042757 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.041956 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.170159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.170125 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx"] Oct 30 20:00:56.191551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.191518 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-eaa5cd-8pzvx"] Oct 30 20:00:56.294867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.294758 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.497820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.497779 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.707474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.707436 5733 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:00:56.708972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.708939 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:00:56.802106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.789918 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.809957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.808048 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 20:00:56.809957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:56.809179 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" containerID="cri-o://12ae1382066279728d6d959ea5d012ea147638af07d101683a46d2e50b37ae93" gracePeriod=30 Oct 30 20:00:57.579080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:57.573707 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.707251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:57.706569 5733 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:00:57.707251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:57.706640 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:00:57.948731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:57.948050 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:58.061307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:58.059992 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1032716-d9e7-4ff9-8504-760423b63c34" path="/var/lib/kubelet/pods/d1032716-d9e7-4ff9-8504-760423b63c34/volumes" Oct 30 20:00:58.678605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:58.678569 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:58.699804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:58.699775 5733 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:00:58.699984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:58.699825 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:00:59.462488 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:00:59.462450 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:00:59.699278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:59.699245 5733 patch_prober.go:29] interesting pod/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Oct 30 20:00:59.699457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:59.699315 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Oct 30 20:00:59.965110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:00:59.960548 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:00.739917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:00.739882 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:01:00.740921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:00.740898 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:01:00.742569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:00.742544 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:01:00.967375 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:00.967327 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.764572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:01.764538 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.928215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:01.928179 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.065297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.065185 5733 generic.go:356] "Generic (PLEG): container finished" podID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerID="12ae1382066279728d6d959ea5d012ea147638af07d101683a46d2e50b37ae93" exitCode=0 Oct 30 20:01:02.076696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.076660 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" event={"ID":"655268c1-99fe-42d1-9545-fa137a8dd02c","Type":"ContainerDied","Data":"12ae1382066279728d6d959ea5d012ea147638af07d101683a46d2e50b37ae93"} Oct 30 20:01:02.584094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.582547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.698484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.698452 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 20:01:02.791035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.790985 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791033 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791080 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791111 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791159 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791192 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791349 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791394 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.791615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.791542 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\") pod \"655268c1-99fe-42d1-9545-fa137a8dd02c\" (UID: \"655268c1-99fe-42d1-9545-fa137a8dd02c\") " Oct 30 20:01:02.792441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.792412 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:02.792627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.792611 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:02.792742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.792727 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:02.793223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.793134 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:02.793529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.793500 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:02.795364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.795334 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:02.795871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.795841 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:02.796005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.795970 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "kube-api-access-hxf8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:02.797283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.797256 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo" pod "655268c1-99fe-42d1-9545-fa137a8dd02c" (UID: "655268c1-99fe-42d1-9545-fa137a8dd02c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:02.893905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893855 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/655268c1-99fe-42d1-9545-fa137a8dd02c-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.893905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893898 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.893905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893913 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893927 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893942 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893955 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893967 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893980 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxf8v\" (UniqueName: \"kubernetes.io/projected/655268c1-99fe-42d1-9545-fa137a8dd02c-kube-api-access-hxf8v\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:02.894199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:02.893996 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/655268c1-99fe-42d1-9545-fa137a8dd02c-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:03.077439 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.077337 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" Oct 30 20:01:03.077439 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.077379 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb" event={"ID":"655268c1-99fe-42d1-9545-fa137a8dd02c","Type":"ContainerDied","Data":"e5c152858637f22665964706d0fcb8a4e6fea6337849b40e6bc8db2d2a101e2f"} Oct 30 20:01:03.077706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.077453 5733 scope.go:122] "RemoveContainer" containerID="12ae1382066279728d6d959ea5d012ea147638af07d101683a46d2e50b37ae93" Oct 30 20:01:03.085113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.085079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.110765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.110728 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 20:01:03.122807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.122773 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-d8jpb"] Oct 30 20:01:03.292455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.291746 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.472909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:03.472871 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.040870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:04.039800 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.070106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:04.070070 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" path="/var/lib/kubelet/pods/655268c1-99fe-42d1-9545-fa137a8dd02c/volumes" Oct 30 20:01:04.850681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:04.850646 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.023715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:05.023675 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.424079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:05.423917 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.649071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:05.649024 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.729078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:06.728036 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.588383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:07.588347 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 20:01:07.588931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:07.588866 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" podUID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" containerName="maas-api" containerID="cri-o://a4f2e35376a2df443eaa9c5042e714cf1106a70c8dac90e172afc9dc51262023" gracePeriod=30 Oct 30 20:01:07.683635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:07.683598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.163418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.155594 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.183751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.183600 5733 generic.go:356] "Generic (PLEG): container finished" podID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" containerID="a4f2e35376a2df443eaa9c5042e714cf1106a70c8dac90e172afc9dc51262023" exitCode=0 Oct 30 20:01:08.183751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.183662 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" event={"ID":"4c46ab70-aa39-46da-ae9d-28abcbaf69ba","Type":"ContainerDied","Data":"a4f2e35376a2df443eaa9c5042e714cf1106a70c8dac90e172afc9dc51262023"} Oct 30 20:01:08.184344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.184308 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z"] Oct 30 20:01:08.197029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.197002 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bdd3b5-nxj7z"] Oct 30 20:01:08.228142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.228107 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 20:01:08.398670 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.398637 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.425368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.425252 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\") pod \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " Oct 30 20:01:08.425553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.425462 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\") pod \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " Oct 30 20:01:08.425553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.425496 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\" (UniqueName: \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\") pod \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " Oct 30 20:01:08.425775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.425751 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\") pod \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\" (UID: \"4c46ab70-aa39-46da-ae9d-28abcbaf69ba\") " Oct 30 20:01:08.430114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.428626 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls" pod "4c46ab70-aa39-46da-ae9d-28abcbaf69ba" (UID: "4c46ab70-aa39-46da-ae9d-28abcbaf69ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:08.432155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.432118 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca" pod "4c46ab70-aa39-46da-ae9d-28abcbaf69ba" (UID: "4c46ab70-aa39-46da-ae9d-28abcbaf69ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:08.432614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.432586 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78" pod "4c46ab70-aa39-46da-ae9d-28abcbaf69ba" (UID: "4c46ab70-aa39-46da-ae9d-28abcbaf69ba"). InnerVolumeSpecName "kube-api-access-vwl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:08.433395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.433368 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs" pod "4c46ab70-aa39-46da-ae9d-28abcbaf69ba" (UID: "4c46ab70-aa39-46da-ae9d-28abcbaf69ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:08.529865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.529822 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:08.529865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.529854 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:08.529865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.529871 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwl78\" (UniqueName: \"kubernetes.io/projected/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-kube-api-access-vwl78\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:08.530183 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.529881 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c46ab70-aa39-46da-ae9d-28abcbaf69ba-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:08.689362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:08.689253 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.193808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.193773 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" Oct 30 20:01:09.194279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.193807 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b" event={"ID":"4c46ab70-aa39-46da-ae9d-28abcbaf69ba","Type":"ContainerDied","Data":"0a5806388198130d9ac59b3e2312d00d8f2981652901701838312b9e55a257db"} Oct 30 20:01:09.194279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.193871 5733 scope.go:122] "RemoveContainer" containerID="a4f2e35376a2df443eaa9c5042e714cf1106a70c8dac90e172afc9dc51262023" Oct 30 20:01:09.245001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.244967 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 20:01:09.252555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.252530 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bdd3b5-ff6c67755-th77b"] Oct 30 20:01:09.943843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:09.943801 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.055867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:10.055827 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" path="/var/lib/kubelet/pods/4c46ab70-aa39-46da-ae9d-28abcbaf69ba/volumes" Oct 30 20:01:10.056537 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:10.056507 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5db55fe-d191-45ce-92d6-7b44dc4c68c3" path="/var/lib/kubelet/pods/b5db55fe-d191-45ce-92d6-7b44dc4c68c3/volumes" Oct 30 20:01:10.900984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:10.900950 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:11.845079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:11.843552 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 20:01:11.845079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:11.843817 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" containerID="cri-o://71ba6b53b27a59815262e6cf7b3f14eb675f697471ee60857e1b3edf2605b157" gracePeriod=30 Oct 30 20:01:12.373787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:12.373745 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:12.398694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:12.398661 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:01:12.419909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:12.419879 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:01:12.735249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:12.735157 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.032216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:13.032051 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.306866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:13.306789 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.620352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:13.620298 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.865080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:13.848388 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.061092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:14.060877 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.661299 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:14.661247 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.586282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:15.586242 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.142631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:16.142599 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 20:01:16.296329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:16.296288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:16.503777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:16.503704 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.201840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.197321 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.282105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.282041 5733 generic.go:356] "Generic (PLEG): container finished" podID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerID="71ba6b53b27a59815262e6cf7b3f14eb675f697471ee60857e1b3edf2605b157" exitCode=0 Oct 30 20:01:17.282371 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.282093 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" event={"ID":"45a1ff27-b5f3-4816-abcf-446f2c4b87e6","Type":"ContainerDied","Data":"71ba6b53b27a59815262e6cf7b3f14eb675f697471ee60857e1b3edf2605b157"} Oct 30 20:01:17.341591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.341562 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 20:01:17.399702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.399662 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" podUID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" containerName="payload-processing" containerID="cri-o://37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00" gracePeriod=25 Oct 30 20:01:17.422347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.421251 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" podUID="16214681-4e48-4e56-bc49-3a9511b2ea66" containerName="payload-pre-processing" containerID="cri-o://1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce" gracePeriod=25 Oct 30 20:01:17.447149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447119 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447354 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447450 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447492 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447547 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447582 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447653 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.447694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447788 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.447978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.447957 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\") pod \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\" (UID: \"45a1ff27-b5f3-4816-abcf-446f2c4b87e6\") " Oct 30 20:01:17.448292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.448264 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.448921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.448892 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.449015 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.448924 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.449152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.449131 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.449727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.449683 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:17.449818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.449782 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.452525 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.452498 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:17.452673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.452649 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "kube-api-access-vbrvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:17.454727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.454690 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:17.454830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.454744 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token" pod "45a1ff27-b5f3-4816-abcf-446f2c4b87e6" (UID: "45a1ff27-b5f3-4816-abcf-446f2c4b87e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:17.561472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561440 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561472 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbrvc\" (UniqueName: \"kubernetes.io/projected/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-kube-api-access-vbrvc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561486 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561500 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561514 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561526 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.561910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.561537 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a1ff27-b5f3-4816-abcf-446f2c4b87e6-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.766718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.766680 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:01:17.826823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.826788 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:01:17.869124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.869091 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\" (UniqueName: \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\") pod \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " Oct 30 20:01:17.869599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.869573 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\") pod \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\" (UID: \"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561\") " Oct 30 20:01:17.870389 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.870363 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume" pod "0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" (UID: "0b4aea87-5fa6-444f-bb9f-a14fcb7f3561"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:17.872655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.872623 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25" pod "0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" (UID: "0b4aea87-5fa6-444f-bb9f-a14fcb7f3561"). InnerVolumeSpecName "kube-api-access-2js25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:17.971224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.971189 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\") pod \"16214681-4e48-4e56-bc49-3a9511b2ea66\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " Oct 30 20:01:17.971680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.971652 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume" pod "16214681-4e48-4e56-bc49-3a9511b2ea66" (UID: "16214681-4e48-4e56-bc49-3a9511b2ea66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:17.971680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.971662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\" (UniqueName: \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\") pod \"16214681-4e48-4e56-bc49-3a9511b2ea66\" (UID: \"16214681-4e48-4e56-bc49-3a9511b2ea66\") " Oct 30 20:01:17.972412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.972388 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.972412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.972410 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2js25\" (UniqueName: \"kubernetes.io/projected/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561-kube-api-access-2js25\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.972597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.972419 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16214681-4e48-4e56-bc49-3a9511b2ea66-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:17.973933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:17.973901 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw" pod "16214681-4e48-4e56-bc49-3a9511b2ea66" (UID: "16214681-4e48-4e56-bc49-3a9511b2ea66"). InnerVolumeSpecName "kube-api-access-zzdhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:18.073956 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.073931 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzdhw\" (UniqueName: \"kubernetes.io/projected/16214681-4e48-4e56-bc49-3a9511b2ea66-kube-api-access-zzdhw\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:18.079164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.079113 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.283996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.283961 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.292571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.292547 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" Oct 30 20:01:18.292732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.292565 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7" event={"ID":"45a1ff27-b5f3-4816-abcf-446f2c4b87e6","Type":"ContainerDied","Data":"f8348a7b09b0045dacfca9e9a44161ea94e741fd3da2c43c2f6b35bd3000971c"} Oct 30 20:01:18.292732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.292626 5733 scope.go:122] "RemoveContainer" containerID="71ba6b53b27a59815262e6cf7b3f14eb675f697471ee60857e1b3edf2605b157" Oct 30 20:01:18.294403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.294375 5733 generic.go:356] "Generic (PLEG): container finished" podID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" containerID="37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00" exitCode=0 Oct 30 20:01:18.294526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.294427 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" Oct 30 20:01:18.294526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.294459 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" event={"ID":"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561","Type":"ContainerDied","Data":"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00"} Oct 30 20:01:18.294526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.294514 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2" event={"ID":"0b4aea87-5fa6-444f-bb9f-a14fcb7f3561","Type":"ContainerDied","Data":"91d08f374eb209179946d62cd8d49e70af5e7af47a1b180b1222df38333985ee"} Oct 30 20:01:18.296233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.296210 5733 generic.go:356] "Generic (PLEG): container finished" podID="16214681-4e48-4e56-bc49-3a9511b2ea66" containerID="1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce" exitCode=0 Oct 30 20:01:18.296394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.296378 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" event={"ID":"16214681-4e48-4e56-bc49-3a9511b2ea66","Type":"ContainerDied","Data":"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce"} Oct 30 20:01:18.296473 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.296422 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" event={"ID":"16214681-4e48-4e56-bc49-3a9511b2ea66","Type":"ContainerDied","Data":"8cbbe74e437aaa98e6de8383c21f36ebde2687cd5339127729fb6b510386abc1"} Oct 30 20:01:18.296542 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.296488 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz" Oct 30 20:01:18.307455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.307432 5733 scope.go:122] "RemoveContainer" containerID="37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00" Oct 30 20:01:18.320540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.320521 5733 scope.go:122] "RemoveContainer" containerID="37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00" Oct 30 20:01:18.320844 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:18.320823 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00\": container with ID starting with 37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00 not found: ID does not exist" containerID="37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00" Oct 30 20:01:18.320958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.320856 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00"} err="failed to get container status \"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00\": rpc error: code = NotFound desc = could not find container \"37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00\": container with ID starting with 37ec2498ed612bea922536a350d3ac73d636be494083efe0619a180664819e00 not found: ID does not exist" Oct 30 20:01:18.320958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.320879 5733 scope.go:122] "RemoveContainer" containerID="1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce" Oct 30 20:01:18.337985 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.337959 5733 scope.go:122] "RemoveContainer" containerID="1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce" Oct 30 20:01:18.342184 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:18.342158 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce\": container with ID starting with 1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce not found: ID does not exist" containerID="1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce" Oct 30 20:01:18.342293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.342196 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce"} err="failed to get container status \"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce\": rpc error: code = NotFound desc = could not find container \"1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce\": container with ID starting with 1144fd4cd791570ae76b288049e81690e971ea26ac1fc0f7864febd3f12b70ce not found: ID does not exist" Oct 30 20:01:18.347968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.347943 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 20:01:18.364905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.364875 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-4dqn7"] Oct 30 20:01:18.402764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.402728 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:01:18.449676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.449640 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae43d70e-77fbd88fc5-c5lfz"] Oct 30 20:01:18.464883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.464849 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:01:18.476169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.476136 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae43d70e-847d9b47f4-rp6t2"] Oct 30 20:01:18.507824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.507791 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 20:01:18.508032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.508007 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" containerID="cri-o://b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1" gracePeriod=30 Oct 30 20:01:18.546909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.546866 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.745681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:18.745639 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.812404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:19.812369 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:01:19.813207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:19.813173 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" podUID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" containerName="maas-api" containerID="cri-o://7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d" gracePeriod=30 Oct 30 20:01:19.816195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:19.816171 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.059014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.058978 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" path="/var/lib/kubelet/pods/0b4aea87-5fa6-444f-bb9f-a14fcb7f3561/volumes" Oct 30 20:01:20.059875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.059849 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16214681-4e48-4e56-bc49-3a9511b2ea66" path="/var/lib/kubelet/pods/16214681-4e48-4e56-bc49-3a9511b2ea66/volumes" Oct 30 20:01:20.060562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.060539 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" path="/var/lib/kubelet/pods/45a1ff27-b5f3-4816-abcf-446f2c4b87e6/volumes" Oct 30 20:01:20.130111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.130081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.189531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.189501 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:01:20.272729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.272696 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\") pod \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " Oct 30 20:01:20.272928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.272766 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\" (UniqueName: \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\") pod \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " Oct 30 20:01:20.272928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.272795 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\") pod \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " Oct 30 20:01:20.272928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.272858 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\") pod \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\" (UID: \"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824\") " Oct 30 20:01:20.273206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.273180 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca" pod "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" (UID: "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:20.273675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.273651 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:20.282441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.282402 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls" pod "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" (UID: "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:20.282441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.282427 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz" pod "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" (UID: "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824"). InnerVolumeSpecName "kube-api-access-87kzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:20.286118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.286086 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs" pod "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" (UID: "4a11fe9c-b2a8-4a85-88f7-d766e1d0e824"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:20.322645 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.322616 5733 generic.go:356] "Generic (PLEG): container finished" podID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" containerID="7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d" exitCode=0 Oct 30 20:01:20.322784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.322671 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" event={"ID":"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824","Type":"ContainerDied","Data":"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d"} Oct 30 20:01:20.322784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.322728 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" event={"ID":"4a11fe9c-b2a8-4a85-88f7-d766e1d0e824","Type":"ContainerDied","Data":"58c2a853981c8461c5e0bf4ef493283cc617b31bf3c28ca0942927670edeaa15"} Oct 30 20:01:20.322784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.322757 5733 scope.go:122] "RemoveContainer" containerID="7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d" Oct 30 20:01:20.323021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.322995 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj" Oct 30 20:01:20.336955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.336931 5733 scope.go:122] "RemoveContainer" containerID="7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d" Oct 30 20:01:20.337398 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:20.337372 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d\": container with ID starting with 7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d not found: ID does not exist" containerID="7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d" Oct 30 20:01:20.337522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.337406 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d"} err="failed to get container status \"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d\": rpc error: code = NotFound desc = could not find container \"7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d\": container with ID starting with 7ad10167ced3023fcb4237e38bd246ec855cb398bd5de69ae9c7399f3534611d not found: ID does not exist" Oct 30 20:01:20.380104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.376706 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87kzz\" (UniqueName: \"kubernetes.io/projected/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-kube-api-access-87kzz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:20.380104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.376746 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:20.380104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.376763 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:20.497207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.492437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.635957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.634820 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:01:20.664930 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.664900 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae43d70e-7bc8c6d859-x9dmj"] Oct 30 20:01:20.665375 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.665337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.818864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:20.818782 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.143926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.143875 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" podUID="16ad9fc8-b478-4513-bf7c-76983ae03126" containerName="payload-pre-processing" containerID="cri-o://b43186f22064504a0ad2f97f66f4e4e915254181c3b0230d074dc779eccf12ab" gracePeriod=25 Oct 30 20:01:21.353958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.353745 5733 generic.go:356] "Generic (PLEG): container finished" podID="16ad9fc8-b478-4513-bf7c-76983ae03126" containerID="b43186f22064504a0ad2f97f66f4e4e915254181c3b0230d074dc779eccf12ab" exitCode=0 Oct 30 20:01:21.353958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.353859 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" event={"ID":"16ad9fc8-b478-4513-bf7c-76983ae03126","Type":"ContainerDied","Data":"b43186f22064504a0ad2f97f66f4e4e915254181c3b0230d074dc779eccf12ab"} Oct 30 20:01:21.448846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.448813 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.467999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.467966 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 20:01:21.628559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.628522 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\") pod \"16ad9fc8-b478-4513-bf7c-76983ae03126\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " Oct 30 20:01:21.628851 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.628830 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\" (UniqueName: \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\") pod \"16ad9fc8-b478-4513-bf7c-76983ae03126\" (UID: \"16ad9fc8-b478-4513-bf7c-76983ae03126\") " Oct 30 20:01:21.628989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.628958 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume" pod "16ad9fc8-b478-4513-bf7c-76983ae03126" (UID: "16ad9fc8-b478-4513-bf7c-76983ae03126"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:21.629736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.629698 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16ad9fc8-b478-4513-bf7c-76983ae03126-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:21.631147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.631116 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l" pod "16ad9fc8-b478-4513-bf7c-76983ae03126" (UID: "16ad9fc8-b478-4513-bf7c-76983ae03126"). InnerVolumeSpecName "kube-api-access-hfl4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:21.750990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:21.750901 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfl4l\" (UniqueName: \"kubernetes.io/projected/16ad9fc8-b478-4513-bf7c-76983ae03126-kube-api-access-hfl4l\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.054595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.054518 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" path="/var/lib/kubelet/pods/4a11fe9c-b2a8-4a85-88f7-d766e1d0e824/volumes" Oct 30 20:01:22.307777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.307688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.371315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.371284 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" Oct 30 20:01:22.371468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.371320 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk" event={"ID":"16ad9fc8-b478-4513-bf7c-76983ae03126","Type":"ContainerDied","Data":"86da29b7ee9bf3b9794f3f3f195d50064f2bec971c197d95b1bb241f5ca3ffa2"} Oct 30 20:01:22.371468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.371383 5733 scope.go:122] "RemoveContainer" containerID="b43186f22064504a0ad2f97f66f4e4e915254181c3b0230d074dc779eccf12ab" Oct 30 20:01:22.399953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.399925 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 20:01:22.407836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:22.407811 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c4c072-658ff9bf7f-kr8qk"] Oct 30 20:01:23.533941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:23.533904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.943376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:23.943347 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.009994 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010138 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010248 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010286 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010302 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010346 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010461 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010533 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010594 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.010657 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\") pod \"cc379c13-37af-4e6c-92dc-5ac7284291c5\" (UID: \"cc379c13-37af-4e6c-92dc-5ac7284291c5\") " Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.011432 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.028417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.011803 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:24.032155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.031242 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:24.032155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.031439 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:24.034106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.032388 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:24.051991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.049355 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "kube-api-access-ksdkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:24.059414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.058877 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16ad9fc8-b478-4513-bf7c-76983ae03126" path="/var/lib/kubelet/pods/16ad9fc8-b478-4513-bf7c-76983ae03126/volumes" Oct 30 20:01:24.068222 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.068176 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:24.068515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.068485 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:24.071657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.071630 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo" pod "cc379c13-37af-4e6c-92dc-5ac7284291c5" (UID: "cc379c13-37af-4e6c-92dc-5ac7284291c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114292 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksdkq\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-kube-api-access-ksdkq\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114329 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114345 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114361 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114376 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc379c13-37af-4e6c-92dc-5ac7284291c5-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114390 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114403 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc379c13-37af-4e6c-92dc-5ac7284291c5-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.114448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.114417 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc379c13-37af-4e6c-92dc-5ac7284291c5-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:24.398126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.398087 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerDied","Data":"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1"} Oct 30 20:01:24.398126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.398121 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" Oct 30 20:01:24.398405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.398159 5733 scope.go:122] "RemoveContainer" containerID="b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1" Oct 30 20:01:24.398405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.398047 5733 generic.go:356] "Generic (PLEG): container finished" podID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerID="b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1" exitCode=0 Oct 30 20:01:24.398405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.398257 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh" event={"ID":"cc379c13-37af-4e6c-92dc-5ac7284291c5","Type":"ContainerDied","Data":"68e0ac5cc46595fce4ed64b9aed7b4d874f27fbbb39106832dbcd217cb2a258b"} Oct 30 20:01:24.411374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.411353 5733 scope.go:122] "RemoveContainer" containerID="6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e" Oct 30 20:01:24.425661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.425639 5733 scope.go:122] "RemoveContainer" containerID="b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1" Oct 30 20:01:24.426015 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:24.425994 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1\": container with ID starting with b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1 not found: ID does not exist" containerID="b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1" Oct 30 20:01:24.426118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.426022 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1"} err="failed to get container status \"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1\": rpc error: code = NotFound desc = could not find container \"b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1\": container with ID starting with b51cac020623af3caebdf54da05421edac4df96b18eadd7e7d4dd417ad8a78b1 not found: ID does not exist" Oct 30 20:01:24.426118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.426037 5733 scope.go:122] "RemoveContainer" containerID="6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e" Oct 30 20:01:24.426416 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:24.426385 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e\": container with ID starting with 6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e not found: ID does not exist" containerID="6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e" Oct 30 20:01:24.426466 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.426420 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e"} err="failed to get container status \"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e\": rpc error: code = NotFound desc = could not find container \"6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e\": container with ID starting with 6f90aa2ed627107cf3126340966696c85fad05485d67ebd25e86257f7c440e1e not found: ID does not exist" Oct 30 20:01:24.450586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.450550 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 20:01:24.738957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:24.738849 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bdd3b5-openshift-default-789fdbc7b5-c78jh"] Oct 30 20:01:25.158455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:25.158093 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.128081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:26.124325 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" path="/var/lib/kubelet/pods/cc379c13-37af-4e6c-92dc-5ac7284291c5/volumes" Oct 30 20:01:26.451285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:26.451193 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.585211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:26.585164 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:26.854144 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:26.853165 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.165543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.165180 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 20:01:27.186151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.184079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:27.300075 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.298813 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.351885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.351151 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.551238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.497141 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.606771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.606237 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 20:01:27.634085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.618288 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.634085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.618997 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:01:27.685120 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.666042 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.780089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.767633 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:27.875189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:27.875138 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:28.147133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.146523 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 20:01:28.354615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.338866 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 20:01:28.354615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.338937 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 20:01:28.354615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.339002 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:01:28.354615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.339212 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" containerID="cri-o://b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728" gracePeriod=30 Oct 30 20:01:28.354615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.339447 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" containerID="cri-o://8f729eb7371d20010dba37714d365141485cb32a4f4957940e9c0626ff59b5ea" gracePeriod=30 Oct 30 20:01:28.505647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.505556 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 20:01:28.519140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.519108 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 20:01:28.655563 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.655521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.845175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:28.845082 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.500682 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:29.500630 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:01:29.873956 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:29.873240 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 20:01:29.894649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:29.894614 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 20:01:30.001317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.001282 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:01:30.004269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004240 5733 cpu_manager.go:414] "containerMap: removing container" podUID="16214681-4e48-4e56-bc49-3a9511b2ea66" containerName="payload-pre-processing" Oct 30 20:01:30.004269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004269 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16214681-4e48-4e56-bc49-3a9511b2ea66" containerName="payload-pre-processing" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004285 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004293 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004304 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" containerName="maas-api" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004311 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" containerName="maas-api" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004345 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004352 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004384 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" containerName="maas-api" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004419 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" containerName="maas-api" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004438 5733 cpu_manager.go:414] "containerMap: removing container" podUID="16ad9fc8-b478-4513-bf7c-76983ae03126" containerName="payload-pre-processing" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004447 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16ad9fc8-b478-4513-bf7c-76983ae03126" containerName="payload-pre-processing" Oct 30 20:01:30.004454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004457 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" containerName="payload-processing" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004464 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" containerName="payload-processing" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004484 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004490 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c972e35-d044-4548-afa9-003279837087" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004517 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004524 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004546 5733 cpu_manager.go:414] "containerMap: removing container" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004553 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257ce18b-20a1-4cd6-9745-47d82cc229ea" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004572 5733 cpu_manager.go:414] "containerMap: removing container" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004581 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004593 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004599 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004612 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004619 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004633 5733 cpu_manager.go:414] "containerMap: removing container" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004640 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004651 5733 cpu_manager.go:414] "containerMap: removing container" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:01:30.004819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004658 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ebb26b-cecd-46fd-a23d-368a78afdf03" containerName="cleanup" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004853 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b553ede-8241-48cb-9d9c-fcb7df9f0a90" containerName="cleanup" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004868 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a11fe9c-b2a8-4a85-88f7-d766e1d0e824" containerName="maas-api" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004883 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="16ad9fc8-b478-4513-bf7c-76983ae03126" containerName="payload-pre-processing" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004893 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4aea87-5fa6-444f-bb9f-a14fcb7f3561" containerName="payload-processing" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004903 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="62521f8f-3368-415c-8e47-d0bc6f967c5d" containerName="cleanup" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004914 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b5b7900-2d3a-44bc-a3e4-08b47b4c98db" containerName="cleanup" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004926 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004937 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="655268c1-99fe-42d1-9545-fa137a8dd02c" containerName="istio-proxy" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004947 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc379c13-37af-4e6c-92dc-5ac7284291c5" containerName="istio-proxy" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004957 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a1ff27-b5f3-4816-abcf-446f2c4b87e6" containerName="istio-proxy" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004966 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="16214681-4e48-4e56-bc49-3a9511b2ea66" containerName="payload-pre-processing" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004975 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e578b6ca-d719-4dcf-8d46-d59825404e14" containerName="cleanup" Oct 30 20:01:30.005461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.004986 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c46ab70-aa39-46da-ae9d-28abcbaf69ba" containerName="maas-api" Oct 30 20:01:30.042081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.020314 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.071105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.056297 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-ed8b50-serving-cert\"" Oct 30 20:01:30.071105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.056904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.071105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.069838 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:01:30.093395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.093345 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhhm\" (UniqueName: \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.093563 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.093404 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.093640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.093606 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.093640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.093629 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.195025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.194945 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhhm\" (UniqueName: \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.195025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.194995 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.195255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.195183 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.195255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.195221 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.195957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.195924 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.197700 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.197676 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.197804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.197747 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.234081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.210391 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.259107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.242729 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhhm\" (UniqueName: \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\") pod \"maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:30.384084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.380844 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.390079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:30.385338 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:31.096492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:31.092219 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.251856 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:01:31.251789 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7854f0d2_ea62_4808_bc07_5bfdd05b55f2.slice/crio-32b150792356f6886360daea565b2f5cd415fdac52dc8270e1b788f4515257a8 WatchSource:0}: Error finding container 32b150792356f6886360daea565b2f5cd415fdac52dc8270e1b788f4515257a8: Status 404 returned error can't find the container with id 32b150792356f6886360daea565b2f5cd415fdac52dc8270e1b788f4515257a8 Oct 30 20:01:31.253067 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:31.253027 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:01:31.649032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:31.648995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.807533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:31.807503 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" event={"ID":"7854f0d2-ea62-4808-bc07-5bfdd05b55f2","Type":"ContainerStarted","Data":"32b150792356f6886360daea565b2f5cd415fdac52dc8270e1b788f4515257a8"} Oct 30 20:01:32.551536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:32.551459 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:32.819255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:32.819156 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" event={"ID":"7854f0d2-ea62-4808-bc07-5bfdd05b55f2","Type":"ContainerStarted","Data":"c60feac2558d6404b9a04e35f405e39089ee73ce1f3c193225d68cde991a26d5"} Oct 30 20:01:32.819255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:32.819241 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:32.851590 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:32.851528 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" podStartSLOduration=3.317532905 podStartE2EDuration="3.851508646s" totalImagesPullingTime="533.975741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:32.844358347 +0000 UTC m=+1852.912193916" watchObservedRunningTime="2026-10-30 20:01:32.851508646 +0000 UTC m=+1852.919344215" Oct 30 20:01:33.506814 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.506772 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" podUID="07e05429-d1e2-424a-8059-3c949fb1a083" containerName="payload-processing" containerID="cri-o://552768eebf3ead751e4620e35da8985fb78caeca73104fbc37b257b2700baebc" gracePeriod=25 Oct 30 20:01:33.520231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.520117 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" podUID="274c9fcd-f20f-4e00-837f-17e92dc7820d" containerName="payload-pre-processing" containerID="cri-o://599e0aa4849ad6a7ad90c3d5ffdc64d3c558fe7979ad8555191b50deed541cbf" gracePeriod=25 Oct 30 20:01:33.750766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.737392 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.750766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.744148 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:01:33.750766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.744204 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:33.814818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.814788 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 20:01:33.840349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.840292 5733 generic.go:356] "Generic (PLEG): container finished" podID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerID="b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728" exitCode=0 Oct 30 20:01:33.840699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.840500 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" event={"ID":"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42","Type":"ContainerDied","Data":"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728"} Oct 30 20:01:33.840699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.840553 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" event={"ID":"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42","Type":"ContainerDied","Data":"8d2e11b9c8b2c4f27576b464220bc75c8c4007a5d035db1b6d1afcbad825d4b4"} Oct 30 20:01:33.840699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.840586 5733 scope.go:122] "RemoveContainer" containerID="b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728" Oct 30 20:01:33.840910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.840753 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc" Oct 30 20:01:33.879827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.879180 5733 generic.go:356] "Generic (PLEG): container finished" podID="07e05429-d1e2-424a-8059-3c949fb1a083" containerID="552768eebf3ead751e4620e35da8985fb78caeca73104fbc37b257b2700baebc" exitCode=0 Oct 30 20:01:33.879827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.879329 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" event={"ID":"07e05429-d1e2-424a-8059-3c949fb1a083","Type":"ContainerDied","Data":"552768eebf3ead751e4620e35da8985fb78caeca73104fbc37b257b2700baebc"} Oct 30 20:01:33.883815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.883549 5733 generic.go:356] "Generic (PLEG): container finished" podID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerID="8f729eb7371d20010dba37714d365141485cb32a4f4957940e9c0626ff59b5ea" exitCode=0 Oct 30 20:01:33.883815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.883741 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" event={"ID":"410ce3f8-0736-4f70-b75d-8e3ed39cbc88","Type":"ContainerDied","Data":"8f729eb7371d20010dba37714d365141485cb32a4f4957940e9c0626ff59b5ea"} Oct 30 20:01:33.887735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.887707 5733 generic.go:356] "Generic (PLEG): container finished" podID="274c9fcd-f20f-4e00-837f-17e92dc7820d" containerID="599e0aa4849ad6a7ad90c3d5ffdc64d3c558fe7979ad8555191b50deed541cbf" exitCode=0 Oct 30 20:01:33.889210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.888530 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" event={"ID":"274c9fcd-f20f-4e00-837f-17e92dc7820d","Type":"ContainerDied","Data":"599e0aa4849ad6a7ad90c3d5ffdc64d3c558fe7979ad8555191b50deed541cbf"} Oct 30 20:01:33.889210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.888616 5733 scope.go:122] "RemoveContainer" containerID="b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728" Oct 30 20:01:33.889465 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:33.889447 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728\": container with ID starting with b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728 not found: ID does not exist" containerID="b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728" Oct 30 20:01:33.889554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.889540 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728"} err="failed to get container status \"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728\": rpc error: code = NotFound desc = could not find container \"b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728\": container with ID starting with b9f078f48c0fd09c33b3175396162aa60cb082d011f66a294be38f90bad81728 not found: ID does not exist" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.987704 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.987761 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.987853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.987913 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988040 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988110 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988158 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988372 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988481 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\") pod \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\" (UID: \"56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42\") " Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.988937 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.989545 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.989774 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.989942 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.990250 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.990980 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.991006 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.991022 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.991039 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:33.992436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.991456 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:33.999822 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.999773 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:33.999822 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:33.999785 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:34.038081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.023678 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.063250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.063119 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p" pod "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" (UID: "56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42"). InnerVolumeSpecName "kube-api-access-qs97p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.078470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.078444 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.091609 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.091704 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.091779 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.091957 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.092026 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.092092 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.092131 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.092185 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.092450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.092237 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\") pod \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\" (UID: \"410ce3f8-0736-4f70-b75d-8e3ed39cbc88\") " Oct 30 20:01:34.093105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093042 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs97p\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-kube-api-access-qs97p\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.093105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093081 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.093105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093097 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.093295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093111 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.093295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093125 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.093693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.093669 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.094918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.094897 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.095439 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.095418 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.095781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.095761 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.095896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.095808 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.118555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.118478 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "kube-api-access-m4qxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.123033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.119473 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.123033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.119831 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:01:34.123033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.120192 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy" pod "410ce3f8-0736-4f70-b75d-8e3ed39cbc88" (UID: "410ce3f8-0736-4f70-b75d-8e3ed39cbc88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:34.126879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.126551 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 20:01:34.187357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.187315 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 20:01:34.193755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.193727 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\" (UniqueName: \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\") pod \"274c9fcd-f20f-4e00-837f-17e92dc7820d\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " Oct 30 20:01:34.193903 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.193857 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\") pod \"274c9fcd-f20f-4e00-837f-17e92dc7820d\" (UID: \"274c9fcd-f20f-4e00-837f-17e92dc7820d\") " Oct 30 20:01:34.194471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194437 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume" pod "274c9fcd-f20f-4e00-837f-17e92dc7820d" (UID: "274c9fcd-f20f-4e00-837f-17e92dc7820d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.194584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194546 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194569 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194584 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194599 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194602 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-z8kpc"] Oct 30 20:01:34.194684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194613 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4qxw\" (UniqueName: \"kubernetes.io/projected/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-kube-api-access-m4qxw\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194668 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194687 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194702 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.194877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.194715 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ce3f8-0736-4f70-b75d-8e3ed39cbc88-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.196998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.196970 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz" pod "274c9fcd-f20f-4e00-837f-17e92dc7820d" (UID: "274c9fcd-f20f-4e00-837f-17e92dc7820d"). InnerVolumeSpecName "kube-api-access-pk2pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.295551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.295458 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\" (UniqueName: \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\") pod \"07e05429-d1e2-424a-8059-3c949fb1a083\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " Oct 30 20:01:34.295746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.295691 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\") pod \"07e05429-d1e2-424a-8059-3c949fb1a083\" (UID: \"07e05429-d1e2-424a-8059-3c949fb1a083\") " Oct 30 20:01:34.296215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.296185 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume" pod "07e05429-d1e2-424a-8059-3c949fb1a083" (UID: "07e05429-d1e2-424a-8059-3c949fb1a083"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:34.296450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.296434 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk2pz\" (UniqueName: \"kubernetes.io/projected/274c9fcd-f20f-4e00-837f-17e92dc7820d-kube-api-access-pk2pz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.296529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.296453 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07e05429-d1e2-424a-8059-3c949fb1a083-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.296529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.296463 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/274c9fcd-f20f-4e00-837f-17e92dc7820d-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.297888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.297854 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt" pod "07e05429-d1e2-424a-8059-3c949fb1a083" (UID: "07e05429-d1e2-424a-8059-3c949fb1a083"). InnerVolumeSpecName "kube-api-access-gxrxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:34.397802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.397764 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrxt\" (UniqueName: \"kubernetes.io/projected/07e05429-d1e2-424a-8059-3c949fb1a083-kube-api-access-gxrxt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:34.575697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.575578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.879808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.879758 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" podUID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" containerName="payload-processing" containerID="cri-o://5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510" gracePeriod=25 Oct 30 20:01:34.895072 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.895020 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" podUID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" containerName="payload-pre-processing" containerID="cri-o://5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3" gracePeriod=25 Oct 30 20:01:34.901898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.901872 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" Oct 30 20:01:34.902011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.901895 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2" event={"ID":"07e05429-d1e2-424a-8059-3c949fb1a083","Type":"ContainerDied","Data":"06606d3d404abdf8fc108dc6ceb89f0678699c1a814e04030756f731b6f3a76d"} Oct 30 20:01:34.902011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.902002 5733 scope.go:122] "RemoveContainer" containerID="552768eebf3ead751e4620e35da8985fb78caeca73104fbc37b257b2700baebc" Oct 30 20:01:34.904268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.903905 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" event={"ID":"410ce3f8-0736-4f70-b75d-8e3ed39cbc88","Type":"ContainerDied","Data":"717404bd8ebc3c8b9fb5010ac1f802b85c829786eac975b0479b06c647539ba3"} Oct 30 20:01:34.904268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.903980 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5" Oct 30 20:01:34.912021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.911994 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" event={"ID":"274c9fcd-f20f-4e00-837f-17e92dc7820d","Type":"ContainerDied","Data":"0ef8fea051521e4a8491fa36e4cb837ff4f4878f5c4d13a40353ddb751bd6897"} Oct 30 20:01:34.912429 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:34.912261 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t" Oct 30 20:01:35.056594 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.056546 5733 scope.go:122] "RemoveContainer" containerID="8f729eb7371d20010dba37714d365141485cb32a4f4957940e9c0626ff59b5ea" Oct 30 20:01:35.072939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.072868 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 20:01:35.082939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.082331 5733 scope.go:122] "RemoveContainer" containerID="599e0aa4849ad6a7ad90c3d5ffdc64d3c558fe7979ad8555191b50deed541cbf" Oct 30 20:01:35.086949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.086926 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-zp2p5"] Oct 30 20:01:35.108279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.108235 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 20:01:35.117525 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.115437 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-98a0d8a9-5f599c6dd7-xw66t"] Oct 30 20:01:35.136315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.136231 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 20:01:35.144138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.144104 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-98a0d8a9-59655bcd4d-jwpt2"] Oct 30 20:01:35.337031 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.337000 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 20:01:35.393412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.393300 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 20:01:35.423514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.423475 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.432254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.429051 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\" (UniqueName: \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\") pod \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " Oct 30 20:01:35.432499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.432411 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\" (UniqueName: \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\") pod \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " Oct 30 20:01:35.432499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.432492 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\") pod \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\" (UID: \"ab3c6bd6-14b6-499f-abf6-99ff8999eee8\") " Oct 30 20:01:35.432694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.432594 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\") pod \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\" (UID: \"0fb1dd9f-fc5b-406b-9a08-769097ed3528\") " Oct 30 20:01:35.433239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.433203 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn" pod "ab3c6bd6-14b6-499f-abf6-99ff8999eee8" (UID: "ab3c6bd6-14b6-499f-abf6-99ff8999eee8"). InnerVolumeSpecName "kube-api-access-mmbrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:35.433617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.433590 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbrn\" (UniqueName: \"kubernetes.io/projected/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-kube-api-access-mmbrn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.433813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.433788 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume" pod "ab3c6bd6-14b6-499f-abf6-99ff8999eee8" (UID: "ab3c6bd6-14b6-499f-abf6-99ff8999eee8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:35.436684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.436652 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k" pod "0fb1dd9f-fc5b-406b-9a08-769097ed3528" (UID: "0fb1dd9f-fc5b-406b-9a08-769097ed3528"). InnerVolumeSpecName "kube-api-access-7qs6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:35.452144 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.451556 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume" pod "0fb1dd9f-fc5b-406b-9a08-769097ed3528" (UID: "0fb1dd9f-fc5b-406b-9a08-769097ed3528"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:35.536427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.536387 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qs6k\" (UniqueName: \"kubernetes.io/projected/0fb1dd9f-fc5b-406b-9a08-769097ed3528-kube-api-access-7qs6k\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.536427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.536428 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3c6bd6-14b6-499f-abf6-99ff8999eee8-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.536684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.536443 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fb1dd9f-fc5b-406b-9a08-769097ed3528-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.929958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.929925 5733 generic.go:356] "Generic (PLEG): container finished" podID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" containerID="5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3" exitCode=0 Oct 30 20:01:35.930476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.929991 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" Oct 30 20:01:35.930476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.930023 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" event={"ID":"ab3c6bd6-14b6-499f-abf6-99ff8999eee8","Type":"ContainerDied","Data":"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3"} Oct 30 20:01:35.930476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.930096 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t" event={"ID":"ab3c6bd6-14b6-499f-abf6-99ff8999eee8","Type":"ContainerDied","Data":"cacec2e14ed65fcbfe88238daf7417795d73645808375cdbf12cc32ad3086eaf"} Oct 30 20:01:35.930476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.930119 5733 scope.go:122] "RemoveContainer" containerID="5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3" Oct 30 20:01:35.937714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.937674 5733 generic.go:356] "Generic (PLEG): container finished" podID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" containerID="5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510" exitCode=0 Oct 30 20:01:35.937852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.937775 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" event={"ID":"0fb1dd9f-fc5b-406b-9a08-769097ed3528","Type":"ContainerDied","Data":"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510"} Oct 30 20:01:35.937852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.937817 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" event={"ID":"0fb1dd9f-fc5b-406b-9a08-769097ed3528","Type":"ContainerDied","Data":"995dd955190deaa7fd259a36ce24f492c58e5deba3bcb1653623e76b9f9742ba"} Oct 30 20:01:35.937972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.937886 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl" Oct 30 20:01:35.952335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.952298 5733 scope.go:122] "RemoveContainer" containerID="5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3" Oct 30 20:01:35.952802 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:35.952773 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3\": container with ID starting with 5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3 not found: ID does not exist" containerID="5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3" Oct 30 20:01:35.952897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.952811 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3"} err="failed to get container status \"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3\": rpc error: code = NotFound desc = could not find container \"5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3\": container with ID starting with 5d03349056c44d1e7b5fbad140f5dc21bb6af27355355d6479e1276a2fb6e2a3 not found: ID does not exist" Oct 30 20:01:35.952897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.952835 5733 scope.go:122] "RemoveContainer" containerID="5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510" Oct 30 20:01:35.977436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.977401 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 20:01:35.979612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.978189 5733 scope.go:122] "RemoveContainer" containerID="5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510" Oct 30 20:01:35.979612 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:35.978616 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510\": container with ID starting with 5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510 not found: ID does not exist" containerID="5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510" Oct 30 20:01:35.979612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.978650 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510"} err="failed to get container status \"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510\": rpc error: code = NotFound desc = could not find container \"5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510\": container with ID starting with 5e4f3e934871fa9d65dbe8929dd96f735dd8d81fac2259c847d31db827e84510 not found: ID does not exist" Oct 30 20:01:35.988359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:35.988333 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-dd1b04-86f6dc8867-8r45t"] Oct 30 20:01:36.005944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.005918 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 20:01:36.022837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.022808 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-dd1b04-7b6d6f58f8-hpdbl"] Oct 30 20:01:36.056752 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.056707 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e05429-d1e2-424a-8059-3c949fb1a083" path="/var/lib/kubelet/pods/07e05429-d1e2-424a-8059-3c949fb1a083/volumes" Oct 30 20:01:36.057461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.057434 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" path="/var/lib/kubelet/pods/0fb1dd9f-fc5b-406b-9a08-769097ed3528/volumes" Oct 30 20:01:36.058047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.058027 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="274c9fcd-f20f-4e00-837f-17e92dc7820d" path="/var/lib/kubelet/pods/274c9fcd-f20f-4e00-837f-17e92dc7820d/volumes" Oct 30 20:01:36.059178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.059157 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" path="/var/lib/kubelet/pods/410ce3f8-0736-4f70-b75d-8e3ed39cbc88/volumes" Oct 30 20:01:36.059863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.059842 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" path="/var/lib/kubelet/pods/56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42/volumes" Oct 30 20:01:36.060470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.060451 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" path="/var/lib/kubelet/pods/ab3c6bd6-14b6-499f-abf6-99ff8999eee8/volumes" Oct 30 20:01:36.572142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:36.567756 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:37.564083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.559824 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:37.964727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.964693 5733 generic.go:356] "Generic (PLEG): container finished" podID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerID="9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a" exitCode=0 Oct 30 20:01:37.964927 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.964771 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerDied","Data":"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a"} Oct 30 20:01:37.965336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.965317 5733 scope.go:122] "RemoveContainer" containerID="9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a" Oct 30 20:01:37.968460 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.968413 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:37.968630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.968586 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:37.968713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:37.968688 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:38.096850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.096823 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:01:38.097604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097583 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" containerName="payload-pre-processing" Oct 30 20:01:38.097604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097602 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" containerName="payload-pre-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097615 5733 cpu_manager.go:414] "containerMap: removing container" podUID="274c9fcd-f20f-4e00-837f-17e92dc7820d" containerName="payload-pre-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097620 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="274c9fcd-f20f-4e00-837f-17e92dc7820d" containerName="payload-pre-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097648 5733 cpu_manager.go:414] "containerMap: removing container" podUID="07e05429-d1e2-424a-8059-3c949fb1a083" containerName="payload-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097653 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e05429-d1e2-424a-8059-3c949fb1a083" containerName="payload-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097665 5733 cpu_manager.go:414] "containerMap: removing container" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097670 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097690 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" containerName="payload-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097695 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" containerName="payload-processing" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097706 5733 cpu_manager.go:414] "containerMap: removing container" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" Oct 30 20:01:38.097780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097711 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097810 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb1dd9f-fc5b-406b-9a08-769097ed3528" containerName="payload-processing" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097820 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e4241a-8f6c-4daa-b6c8-3d7dc7cb2e42" containerName="istio-proxy" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097827 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3c6bd6-14b6-499f-abf6-99ff8999eee8" containerName="payload-pre-processing" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097833 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="274c9fcd-f20f-4e00-837f-17e92dc7820d" containerName="payload-pre-processing" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097842 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e05429-d1e2-424a-8059-3c949fb1a083" containerName="payload-processing" Oct 30 20:01:38.098318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.097849 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="410ce3f8-0736-4f70-b75d-8e3ed39cbc88" containerName="istio-proxy" Oct 30 20:01:38.100569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.100551 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.110139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.110114 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c50e1e-openshift-default-dockercfg-wkgrj\"" Oct 30 20:01:38.138680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.137988 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:01:38.194454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194291 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.194698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194658 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.194793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194731 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp6h6\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.194793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194770 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.194907 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194820 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.194984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.194958 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.195048 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.195033 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.195189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.195162 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.195333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.195313 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.296983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.296898 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.296983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.296957 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297013 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297112 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297159 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297243 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297308 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297420 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297552 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297590 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297636 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp6h6\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.297804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.297779 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.298097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.298039 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.298221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.298166 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.299684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.299661 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.299870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.299831 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.309872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.309845 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.316318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.316267 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp6h6\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\") pod \"e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.417448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.417314 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:38.647530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.647493 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:38.733746 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:01:38.733701 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f0f0fc7_8645_4720_86f7_f628266648c8.slice/crio-db5fd12a40bfc0e6c7eab65500f97c0b77b5433e647ae1b4e92d37bb1f1e162d WatchSource:0}: Error finding container db5fd12a40bfc0e6c7eab65500f97c0b77b5433e647ae1b4e92d37bb1f1e162d: Status 404 returned error can't find the container with id db5fd12a40bfc0e6c7eab65500f97c0b77b5433e647ae1b4e92d37bb1f1e162d Oct 30 20:01:38.734319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.734288 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:01:38.737887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.737849 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:38.738115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.738089 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:38.738272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.738253 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:38.894354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.894221 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:01:38.985261 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.982286 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerStarted","Data":"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d"} Oct 30 20:01:38.987610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.985855 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" event={"ID":"4f0f0fc7-8645-4720-86f7-f628266648c8","Type":"ContainerStarted","Data":"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311"} Oct 30 20:01:38.987610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:38.985899 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" event={"ID":"4f0f0fc7-8645-4720-86f7-f628266648c8","Type":"ContainerStarted","Data":"db5fd12a40bfc0e6c7eab65500f97c0b77b5433e647ae1b4e92d37bb1f1e162d"} Oct 30 20:01:39.104620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.104567 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podStartSLOduration=1.104552089 podStartE2EDuration="1.104552089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:39.097027548 +0000 UTC m=+1859.164863108" watchObservedRunningTime="2026-10-30 20:01:39.104552089 +0000 UTC m=+1859.172387657" Oct 30 20:01:39.280523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.280442 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:01:39.280787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.280764 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:39.280856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.280824 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:39.428242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.418173 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:39.428242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.423148 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:39.428242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.423202 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:39.899873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:39.899832 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.281530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.281434 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:40.281530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.281494 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:40.434681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.434642 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:40.434969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.434937 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:40.780476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.780416 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:01:40.789994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.789426 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.797218 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.796922 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.797401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.797261 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzlz\" (UniqueName: \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.797401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.797335 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.797569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.797488 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.806561 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.806535 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-96ec93-serving-cert\"" Oct 30 20:01:40.884083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.879832 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:01:40.894095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.893781 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.899193 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzlz\" (UniqueName: \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.899282 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.899400 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.899479 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:40.899736 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-serving-cert: secret "maas-api-e2e-api-b-96ec93-serving-cert" not found Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:40.899808 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls podName:b46ecfd4-5a1b-4270-8712-c3ff7be83f03 nodeName:}" failed. No retries permitted until 2026-10-30 20:01:41.39978469 +0000 UTC m=+1861.467620238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls") pod "maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" (UID: "b46ecfd4-5a1b-4270-8712-c3ff7be83f03") : secret "maas-api-e2e-api-b-96ec93-serving-cert" not found Oct 30 20:01:40.901171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:40.900923 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:41.063161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.062684 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:41.135083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.094764 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.145082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.141645 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzlz\" (UniqueName: \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:41.296947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.296753 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:41.297244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.297210 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.298276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.298236 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:41.532081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.503035 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:41.532081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.503154 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:41.532081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.518309 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:41.557080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.545668 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") pod \"maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:41.774872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:41.773928 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:42.291227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.288888 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:42.291227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.288945 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:42.426347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.426210 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:42.426347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.426270 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:42.464789 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.464747 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.497778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:42.497721 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:01:42.590218 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:01:42.590174 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46ecfd4_5a1b_4270_8712_c3ff7be83f03.slice/crio-609ab4ac9fc902505458a3ffda24d6584a0e1cb6fabeda731af9681e182cc25f WatchSource:0}: Error finding container 609ab4ac9fc902505458a3ffda24d6584a0e1cb6fabeda731af9681e182cc25f: Status 404 returned error can't find the container with id 609ab4ac9fc902505458a3ffda24d6584a0e1cb6fabeda731af9681e182cc25f Oct 30 20:01:43.281216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.281157 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:43.285566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.285441 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:43.304423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.304396 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" event={"ID":"b46ecfd4-5a1b-4270-8712-c3ff7be83f03","Type":"ContainerStarted","Data":"609ab4ac9fc902505458a3ffda24d6584a0e1cb6fabeda731af9681e182cc25f"} Oct 30 20:01:43.431464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.431437 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:43.431565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.431495 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:43.522085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:43.521768 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.280647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.280612 5733 patch_prober.go:29] interesting pod/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 30 20:01:44.280852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.280787 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 30 20:01:44.363083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.357481 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" event={"ID":"b46ecfd4-5a1b-4270-8712-c3ff7be83f03","Type":"ContainerStarted","Data":"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9"} Oct 30 20:01:44.363083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.358859 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:44.397564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.395617 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.429082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.418504 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:44.429082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:44.418573 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:45.275171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.275103 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" podStartSLOduration=4.65522565 podStartE2EDuration="5.275081054s" totalImagesPullingTime="619.855404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:44.395679064 +0000 UTC m=+1864.463514633" watchObservedRunningTime="2026-10-30 20:01:45.275081054 +0000 UTC m=+1865.342916624" Oct 30 20:01:45.276351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.276324 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 20:01:45.276611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.276583 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" podUID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" containerName="maas-api" containerID="cri-o://8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a" gracePeriod=30 Oct 30 20:01:45.288426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.288383 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:01:45.289828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.289685 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:01:45.291095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.291026 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:01:45.353155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.353117 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.449235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.448647 5733 patch_prober.go:29] interesting pod/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 20:01:45.449235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.448708 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 20:01:45.695605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.677862 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.853575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.853537 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:45.968355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:45.968321 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 20:01:46.003122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.002639 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\") pod \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " Oct 30 20:01:46.003122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.002695 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\" (UniqueName: \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\") pod \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " Oct 30 20:01:46.003122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.002948 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") pod \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " Oct 30 20:01:46.003122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.002986 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\") pod \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\" (UID: \"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5\") " Oct 30 20:01:46.009029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.007418 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca" pod "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" (UID: "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:46.009915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.009886 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls" pod "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" (UID: "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:46.010116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.009971 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs" pod "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" (UID: "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:46.010279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.010264 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw" pod "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" (UID: "b9bc9338-e466-4f64-a5ce-5cc163ac9ca5"). InnerVolumeSpecName "kube-api-access-49gqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:46.106916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.106878 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:46.106916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.106909 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49gqw\" (UniqueName: \"kubernetes.io/projected/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-kube-api-access-49gqw\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:46.106916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.106923 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:46.107293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.106935 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:46.184378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.184342 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.376654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.376620 5733 generic.go:356] "Generic (PLEG): container finished" podID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" containerID="8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a" exitCode=0 Oct 30 20:01:46.376830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.376671 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" Oct 30 20:01:46.376830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.376699 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" event={"ID":"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5","Type":"ContainerDied","Data":"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a"} Oct 30 20:01:46.376830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.376750 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn" event={"ID":"b9bc9338-e466-4f64-a5ce-5cc163ac9ca5","Type":"ContainerDied","Data":"6b0ee1e5a8ddb771e1729d2f154a06367cc28e388e5e22b73b9309707161837e"} Oct 30 20:01:46.376830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.376772 5733 scope.go:122] "RemoveContainer" containerID="8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a" Oct 30 20:01:46.389622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.389592 5733 scope.go:122] "RemoveContainer" containerID="8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a" Oct 30 20:01:46.389923 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:46.389902 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a\": container with ID starting with 8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a not found: ID does not exist" containerID="8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a" Oct 30 20:01:46.389989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.389934 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a"} err="failed to get container status \"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a\": rpc error: code = NotFound desc = could not find container \"8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a\": container with ID starting with 8c356b2e7e5d04cbc64ae7c9d5573237b5e01c41960c7465118f40eded39d77a not found: ID does not exist" Oct 30 20:01:46.423920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.423896 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:46.424253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.424227 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:46.424980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.424959 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:01:46.425210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.425189 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 20:01:46.450748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.450712 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-98a0d8a9-5cc4479596-mlpvn"] Oct 30 20:01:46.552964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:46.552927 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.419132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:47.419090 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.053071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:48.053027 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" path="/var/lib/kubelet/pods/b9bc9338-e466-4f64-a5ce-5cc163ac9ca5/volumes" Oct 30 20:01:48.454197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:48.454160 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.545781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:49.545748 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.438708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:50.438675 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.383579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:51.383534 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:01:51.491984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:51.491954 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.960596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:51.960562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.838332 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:52.838286 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:53.109797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:53.109763 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.239523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:54.239480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.477223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:55.477169 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.694377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:55.694329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.919306 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:55.919268 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.178874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:56.178790 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.801599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:56.801562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.981862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:56.981821 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.384856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.384825 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:01:57.385756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.385721 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" containerName="maas-api" Oct 30 20:01:57.385756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.385742 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" containerName="maas-api" Oct 30 20:01:57.385947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.385871 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9bc9338-e466-4f64-a5ce-5cc163ac9ca5" containerName="maas-api" Oct 30 20:01:57.389838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.389818 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.398184 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.398161 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn\"" Oct 30 20:01:57.413547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.413520 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:01:57.483663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.483625 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.483921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.483902 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.484155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.484133 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.484354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.484333 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4q8\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.484493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.484475 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.484720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.484701 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.484933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.484883 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.486705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.485114 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.486947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.486805 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.518137 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.513187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.710853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.710756 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.719078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.711200 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.719669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.719643 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4q8\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.719866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.719849 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.727292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.720100 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.744254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.744212 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.744573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.744548 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.744889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.744869 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.745120 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.745099 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.749171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.745669 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.766112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.738786 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.766112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.737739 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.766112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.750558 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.766112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.757118 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.777276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.777242 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:57.789157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.784235 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.789157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.786601 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.789157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.787710 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:57.789157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:57.788935 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4q8\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\") pod \"e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:58.010033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.009935 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:58.132265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.127163 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.396358 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:01:58.396321 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34c469c_95a6_4134_b889_d20cbe59c1a3.slice/crio-e7d1da581dc1df5fbc6bbc711ba66233a61b324afb1c6ed06f746a046b7322f2 WatchSource:0}: Error finding container e7d1da581dc1df5fbc6bbc711ba66233a61b324afb1c6ed06f746a046b7322f2: Status 404 returned error can't find the container with id e7d1da581dc1df5fbc6bbc711ba66233a61b324afb1c6ed06f746a046b7322f2 Oct 30 20:01:58.398621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.398589 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:01:58.399789 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.399748 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:58.399900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.399844 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:58.399900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.399881 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:01:58.713429 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.713321 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerStarted","Data":"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3"} Oct 30 20:01:58.713753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.713706 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerStarted","Data":"e7d1da581dc1df5fbc6bbc711ba66233a61b324afb1c6ed06f746a046b7322f2"} Oct 30 20:01:58.750119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:58.750040 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podStartSLOduration=1.750019835 podStartE2EDuration="1.750019835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:58.743655456 +0000 UTC m=+1878.811491026" watchObservedRunningTime="2026-10-30 20:01:58.750019835 +0000 UTC m=+1878.817855404" Oct 30 20:01:59.011279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:59.011188 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:01:59.013440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:59.013413 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:01:59.013695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:59.013654 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:01:59.167121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:01:59.166299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.379113 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:01:59.379079 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:02:00.016136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:00.016102 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:02:00.327132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:00.326182 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.738863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:00.738826 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:02:00.740104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:00.740046 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:02:01.119918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:01.119881 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.388856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:01.388759 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.592549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:01.592503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:02.389661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.389619 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:02.442296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.442251 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:02:02.446559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.446527 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.462484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.462457 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f246b\"" Oct 30 20:02:02.467467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.467432 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:02:02.472230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.472200 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.473694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.473668 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:02:02.494085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.494032 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f246b-dockercfg-8chtl\"" Oct 30 20:02:02.494754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.494726 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.495751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.495717 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbzr\" (UniqueName: \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.500957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.500873 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:02:02.584621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.584550 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:02.602187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.599503 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.626086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.602502 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.626394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.626363 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9wf\" (UniqueName: \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.626724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.626705 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbzr\" (UniqueName: \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.669148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.613016 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.736112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.735013 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.736112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.735136 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9wf\" (UniqueName: \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.775633 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.736802 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.821383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.821341 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbzr\" (UniqueName: \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\") pod \"payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:02.880732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.876853 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9wf\" (UniqueName: \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\") pod \"payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:02.882763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:02.882514 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:03.071148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.071104 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:03.290684 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:02:03.290641 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc31087c_8882_44ff_9240_42cf43576885.slice/crio-ed94facc5056f06a1da49c606c079851df28c8d03d86d10e799a309970a0a479 WatchSource:0}: Error finding container ed94facc5056f06a1da49c606c079851df28c8d03d86d10e799a309970a0a479: Status 404 returned error can't find the container with id ed94facc5056f06a1da49c606c079851df28c8d03d86d10e799a309970a0a479 Oct 30 20:02:03.292463 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.292428 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:02:03.421966 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:02:03.421923 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b80ef9e_12fc_48a1_a912_891358436d57.slice/crio-22b5deb72a3f126d7bf52808efb65ce28498b07c9b0d02721a1bcac1d35e2fa5 WatchSource:0}: Error finding container 22b5deb72a3f126d7bf52808efb65ce28498b07c9b0d02721a1bcac1d35e2fa5: Status 404 returned error can't find the container with id 22b5deb72a3f126d7bf52808efb65ce28498b07c9b0d02721a1bcac1d35e2fa5 Oct 30 20:02:03.423193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.423168 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:02:03.764228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.764130 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:02:03.765345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.765133 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:02:03.868161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.868121 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" event={"ID":"4b80ef9e-12fc-48a1-a912-891358436d57","Type":"ContainerStarted","Data":"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f"} Oct 30 20:02:03.868335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.868191 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:03.868335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.868214 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" event={"ID":"4b80ef9e-12fc-48a1-a912-891358436d57","Type":"ContainerStarted","Data":"22b5deb72a3f126d7bf52808efb65ce28498b07c9b0d02721a1bcac1d35e2fa5"} Oct 30 20:02:03.869814 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.869786 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" event={"ID":"dc31087c-8882-44ff-9240-42cf43576885","Type":"ContainerStarted","Data":"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461"} Oct 30 20:02:03.869960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.869827 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" event={"ID":"dc31087c-8882-44ff-9240-42cf43576885","Type":"ContainerStarted","Data":"ed94facc5056f06a1da49c606c079851df28c8d03d86d10e799a309970a0a479"} Oct 30 20:02:03.870521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.870500 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:03.909311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.909254 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" podStartSLOduration=1.909238555 podStartE2EDuration="1.909238555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:03.904375975 +0000 UTC m=+1883.972211545" watchObservedRunningTime="2026-10-30 20:02:03.909238555 +0000 UTC m=+1883.977074124" Oct 30 20:02:03.955606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.955566 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.958616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:03.958552 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" podStartSLOduration=1.958533423 podStartE2EDuration="1.958533423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:03.938538734 +0000 UTC m=+1884.006374303" watchObservedRunningTime="2026-10-30 20:02:03.958533423 +0000 UTC m=+1884.026368992" Oct 30 20:02:04.130411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:04.130365 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.459157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:04.459046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.839264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:04.839185 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.036868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:05.036835 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.408266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:05.408230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.975323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:05.975290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.490949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:06.490916 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.851911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:06.851878 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.094468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:07.094435 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.267340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:07.267140 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.525585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:07.525469 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.999485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:07.999446 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.197434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:08.197404 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.764837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:08.764784 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" podUID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" containerName="payload-processing" containerID="cri-o://7339e929cfcd01788e4a9b7c06a66f1390dcce9223ef2672e1bc374d3f7af6e2" gracePeriod=25 Oct 30 20:02:08.766358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:08.766321 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" podUID="f43cb101-2fd1-4418-ac86-3920dc32477b" containerName="payload-pre-processing" containerID="cri-o://989985e86c6f9b4c492f8a1aaeb861709b899198d95f41ec3fedd4e5361e0d30" gracePeriod=25 Oct 30 20:02:09.538084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.531592 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.660469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.660420 5733 generic.go:356] "Generic (PLEG): container finished" podID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" containerID="7339e929cfcd01788e4a9b7c06a66f1390dcce9223ef2672e1bc374d3f7af6e2" exitCode=0 Oct 30 20:02:09.661043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.661013 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" event={"ID":"4ad346e3-51bb-493b-a76f-71a08cca6a6b","Type":"ContainerDied","Data":"7339e929cfcd01788e4a9b7c06a66f1390dcce9223ef2672e1bc374d3f7af6e2"} Oct 30 20:02:09.684406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.684308 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:02:09.802605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.794689 5733 generic.go:356] "Generic (PLEG): container finished" podID="f43cb101-2fd1-4418-ac86-3920dc32477b" containerID="989985e86c6f9b4c492f8a1aaeb861709b899198d95f41ec3fedd4e5361e0d30" exitCode=0 Oct 30 20:02:09.802605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.794791 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" event={"ID":"f43cb101-2fd1-4418-ac86-3920dc32477b","Type":"ContainerDied","Data":"989985e86c6f9b4c492f8a1aaeb861709b899198d95f41ec3fedd4e5361e0d30"} Oct 30 20:02:09.945562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.945522 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\" (UniqueName: \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\") pod \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " Oct 30 20:02:09.945723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.945621 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\") pod \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\" (UID: \"4ad346e3-51bb-493b-a76f-71a08cca6a6b\") " Oct 30 20:02:09.946767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.946726 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume" pod "4ad346e3-51bb-493b-a76f-71a08cca6a6b" (UID: "4ad346e3-51bb-493b-a76f-71a08cca6a6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:09.955400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.955263 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7" pod "4ad346e3-51bb-493b-a76f-71a08cca6a6b" (UID: "4ad346e3-51bb-493b-a76f-71a08cca6a6b"). InnerVolumeSpecName "kube-api-access-mmrz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:09.993324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:09.993289 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:02:10.049002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.048959 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmrz7\" (UniqueName: \"kubernetes.io/projected/4ad346e3-51bb-493b-a76f-71a08cca6a6b-kube-api-access-mmrz7\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.049002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.049001 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ad346e3-51bb-493b-a76f-71a08cca6a6b-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.150484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.150441 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\") pod \"f43cb101-2fd1-4418-ac86-3920dc32477b\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " Oct 30 20:02:10.151035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.150998 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume" pod "f43cb101-2fd1-4418-ac86-3920dc32477b" (UID: "f43cb101-2fd1-4418-ac86-3920dc32477b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:10.151372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.151343 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\" (UniqueName: \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\") pod \"f43cb101-2fd1-4418-ac86-3920dc32477b\" (UID: \"f43cb101-2fd1-4418-ac86-3920dc32477b\") " Oct 30 20:02:10.153360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.153332 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f43cb101-2fd1-4418-ac86-3920dc32477b-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.155608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.155580 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n" pod "f43cb101-2fd1-4418-ac86-3920dc32477b" (UID: "f43cb101-2fd1-4418-ac86-3920dc32477b"). InnerVolumeSpecName "kube-api-access-xnx8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:10.254768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.254731 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnx8n\" (UniqueName: \"kubernetes.io/projected/f43cb101-2fd1-4418-ac86-3920dc32477b-kube-api-access-xnx8n\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.399577 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.399540 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:10.804861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.804768 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" Oct 30 20:02:10.804861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.804787 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj" event={"ID":"4ad346e3-51bb-493b-a76f-71a08cca6a6b","Type":"ContainerDied","Data":"9ffd4bf2237dce2f8d6e30cd0d0c426ae276f6983fe9dd044a954f9de62a6250"} Oct 30 20:02:10.804861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.804853 5733 scope.go:122] "RemoveContainer" containerID="7339e929cfcd01788e4a9b7c06a66f1390dcce9223ef2672e1bc374d3f7af6e2" Oct 30 20:02:10.806649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.806622 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" Oct 30 20:02:10.806649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.806637 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq" event={"ID":"f43cb101-2fd1-4418-ac86-3920dc32477b","Type":"ContainerDied","Data":"f1ee1a39c6b6d304425905043e96d010a6962373e166e9f9be06bbdaf6dbe38e"} Oct 30 20:02:10.820345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.820321 5733 scope.go:122] "RemoveContainer" containerID="989985e86c6f9b4c492f8a1aaeb861709b899198d95f41ec3fedd4e5361e0d30" Oct 30 20:02:10.830157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.830129 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:02:10.838417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.838393 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-35deec-66f64c4f64-bkbmj"] Oct 30 20:02:10.860005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.859967 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:02:10.865020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:10.864990 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-35deec-84b665df47-tnrvq"] Oct 30 20:02:11.357078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:11.353522 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.053768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:12.053738 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" path="/var/lib/kubelet/pods/4ad346e3-51bb-493b-a76f-71a08cca6a6b/volumes" Oct 30 20:02:12.054314 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:12.054297 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43cb101-2fd1-4418-ac86-3920dc32477b" path="/var/lib/kubelet/pods/f43cb101-2fd1-4418-ac86-3920dc32477b/volumes" Oct 30 20:02:12.813811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:12.813773 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.080412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:13.080299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.294967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:13.294913 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.511153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:13.511112 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.343357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:14.343319 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.728544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:14.728504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.929480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:14.929451 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:02:15.065420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:15.065321 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.790759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:15.790717 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.965248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:15.965218 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:02:16.379245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:16.379211 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:16.588845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:16.588805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:17.016640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:17.016599 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:17.884408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:17.884367 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:18.142445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:18.142351 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:18.411550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:18.411448 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:18.614680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:18.614643 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:02:18.614898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:18.614865 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" containerID="cri-o://7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53" gracePeriod=30 Oct 30 20:02:18.752278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:18.750999 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.106351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:19.106315 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.433417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:19.433310 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.635580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:19.635536 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.092912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:20.092880 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.361111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:20.357618 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.917636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:20.917538 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.267131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:21.237578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.775560 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:21.775524 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:02:21.788844 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:21.788817 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:02:21.892243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:21.892201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:22.062167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:22.062081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.061077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.061028 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.938348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.938323 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:02:23.972646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.972610 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.972988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.972699 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.972988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.972793 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.972988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.972929 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.973336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.973125 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.973336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.973204 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.973336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.973256 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.973489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.973341 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.973535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.973522 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\") pod \"8dd82080-4fc3-44b6-8554-f0548d18605c\" (UID: \"8dd82080-4fc3-44b6-8554-f0548d18605c\") " Oct 30 20:02:23.976509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.976480 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.977768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.977357 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.977768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.977683 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:23.977768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.977742 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.978345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.978308 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:23.979449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.978663 5733 generic.go:356] "Generic (PLEG): container finished" podID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerID="7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53" exitCode=0 Oct 30 20:02:23.979449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.978777 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" event={"ID":"8dd82080-4fc3-44b6-8554-f0548d18605c","Type":"ContainerDied","Data":"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53"} Oct 30 20:02:23.979449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.978831 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" event={"ID":"8dd82080-4fc3-44b6-8554-f0548d18605c","Type":"ContainerDied","Data":"b411d2a4e9eda17754e5f4cc4c20378079410e8850a9689a58fa8b9ba9d0e332"} Oct 30 20:02:23.979449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.978857 5733 scope.go:122] "RemoveContainer" containerID="7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53" Oct 30 20:02:23.979449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.979020 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw" Oct 30 20:02:23.979775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.979692 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:23.980607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.980584 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:23.981896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.981870 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "kube-api-access-6r9fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:23.989844 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:23.989818 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy" pod "8dd82080-4fc3-44b6-8554-f0548d18605c" (UID: "8dd82080-4fc3-44b6-8554-f0548d18605c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:24.039113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.039093 5733 scope.go:122] "RemoveContainer" containerID="7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53" Oct 30 20:02:24.039438 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:02:24.039395 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53\": container with ID starting with 7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53 not found: ID does not exist" containerID="7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53" Oct 30 20:02:24.039564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.039447 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53"} err="failed to get container status \"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53\": rpc error: code = NotFound desc = could not find container \"7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53\": container with ID starting with 7d3d0f27a160ebf26c609d4124ead20ebd6efdebbc5312db6a6dc19773e43a53 not found: ID does not exist" Oct 30 20:02:24.076747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076716 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd82080-4fc3-44b6-8554-f0548d18605c-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076749 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076765 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076778 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076792 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076805 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r9fl\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-kube-api-access-6r9fl\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076819 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd82080-4fc3-44b6-8554-f0548d18605c-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076832 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.076893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.076845 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd82080-4fc3-44b6-8554-f0548d18605c-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:24.261413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.260872 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:24.325948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.325915 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:02:24.364947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.364913 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dd1b04-openshift-default-5bfd85ddbf-pztmw"] Oct 30 20:02:24.795222 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:24.795187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:25.833540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:25.833506 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.085156 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:26.082606 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.171072 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:26.171026 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" path="/var/lib/kubelet/pods/8dd82080-4fc3-44b6-8554-f0548d18605c/volumes" Oct 30 20:02:26.776835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:26.776792 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" podUID="85a663b2-8dd3-4532-9269-a6048f484290" containerName="payload-processing" containerID="cri-o://2ac925cb55703aab8e717fa2e268134868cd38fded682df87e7288cb7597ad91" gracePeriod=25 Oct 30 20:02:26.789651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:26.789561 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" podUID="b0c0ab13-e06c-433e-995c-ba78139f9da1" containerName="payload-pre-processing" containerID="cri-o://ec7ba420ef29488248eeea9f351a7c0249cd866b9c8d71d71a535f547e25d95c" gracePeriod=25 Oct 30 20:02:27.083122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.080751 5733 generic.go:356] "Generic (PLEG): container finished" podID="b0c0ab13-e06c-433e-995c-ba78139f9da1" containerID="ec7ba420ef29488248eeea9f351a7c0249cd866b9c8d71d71a535f547e25d95c" exitCode=0 Oct 30 20:02:27.083122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.080844 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" event={"ID":"b0c0ab13-e06c-433e-995c-ba78139f9da1","Type":"ContainerDied","Data":"ec7ba420ef29488248eeea9f351a7c0249cd866b9c8d71d71a535f547e25d95c"} Oct 30 20:02:27.083732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.083126 5733 generic.go:356] "Generic (PLEG): container finished" podID="85a663b2-8dd3-4532-9269-a6048f484290" containerID="2ac925cb55703aab8e717fa2e268134868cd38fded682df87e7288cb7597ad91" exitCode=0 Oct 30 20:02:27.083732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.083304 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" event={"ID":"85a663b2-8dd3-4532-9269-a6048f484290","Type":"ContainerDied","Data":"2ac925cb55703aab8e717fa2e268134868cd38fded682df87e7288cb7597ad91"} Oct 30 20:02:27.175614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.175579 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:02:27.196269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.196246 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:02:27.200905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.200876 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\" (UniqueName: \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\") pod \"85a663b2-8dd3-4532-9269-a6048f484290\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " Oct 30 20:02:27.201040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.200933 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\" (UniqueName: \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\") pod \"b0c0ab13-e06c-433e-995c-ba78139f9da1\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " Oct 30 20:02:27.201040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.201013 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\") pod \"85a663b2-8dd3-4532-9269-a6048f484290\" (UID: \"85a663b2-8dd3-4532-9269-a6048f484290\") " Oct 30 20:02:27.201201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.201106 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\") pod \"b0c0ab13-e06c-433e-995c-ba78139f9da1\" (UID: \"b0c0ab13-e06c-433e-995c-ba78139f9da1\") " Oct 30 20:02:27.202134 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.201975 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume" pod "85a663b2-8dd3-4532-9269-a6048f484290" (UID: "85a663b2-8dd3-4532-9269-a6048f484290"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:27.202243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.202134 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume" pod "b0c0ab13-e06c-433e-995c-ba78139f9da1" (UID: "b0c0ab13-e06c-433e-995c-ba78139f9da1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:27.205169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.205143 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g" pod "85a663b2-8dd3-4532-9269-a6048f484290" (UID: "85a663b2-8dd3-4532-9269-a6048f484290"). InnerVolumeSpecName "kube-api-access-t9k9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:27.206663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.206639 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4" pod "b0c0ab13-e06c-433e-995c-ba78139f9da1" (UID: "b0c0ab13-e06c-433e-995c-ba78139f9da1"). InnerVolumeSpecName "kube-api-access-5kxb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:27.303692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.303652 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9k9g\" (UniqueName: \"kubernetes.io/projected/85a663b2-8dd3-4532-9269-a6048f484290-kube-api-access-t9k9g\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:27.303692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.303681 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxb4\" (UniqueName: \"kubernetes.io/projected/b0c0ab13-e06c-433e-995c-ba78139f9da1-kube-api-access-5kxb4\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:27.303692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.303691 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85a663b2-8dd3-4532-9269-a6048f484290-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:27.303692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.303700 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0c0ab13-e06c-433e-995c-ba78139f9da1-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:27.444506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:27.444465 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.093248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.093218 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" Oct 30 20:02:28.093248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.093231 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st" event={"ID":"b0c0ab13-e06c-433e-995c-ba78139f9da1","Type":"ContainerDied","Data":"1aa24a4a3ec62e0d388e535ae71b20428a0383e247a8b15c7cee9e18859a5a3a"} Oct 30 20:02:28.093708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.093328 5733 scope.go:122] "RemoveContainer" containerID="ec7ba420ef29488248eeea9f351a7c0249cd866b9c8d71d71a535f547e25d95c" Oct 30 20:02:28.094929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.094906 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" Oct 30 20:02:28.095035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.094932 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn" event={"ID":"85a663b2-8dd3-4532-9269-a6048f484290","Type":"ContainerDied","Data":"fd88937b9b32543f2a2a9b68d7bbd9d5a39099b905ead385adb5c8e1da04cd56"} Oct 30 20:02:28.104752 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.104732 5733 scope.go:122] "RemoveContainer" containerID="2ac925cb55703aab8e717fa2e268134868cd38fded682df87e7288cb7597ad91" Oct 30 20:02:28.125489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.125460 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:02:28.130237 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.130214 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c50e1e-78c6b8c964-82kkn"] Oct 30 20:02:28.152297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.152268 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:02:28.159491 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.159462 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c50e1e-65677fcccf-c57st"] Oct 30 20:02:28.657447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:28.657410 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.378551 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:02:29.378512 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:02:29.744407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:29.744005 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:30.056553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:30.056470 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85a663b2-8dd3-4532-9269-a6048f484290" path="/var/lib/kubelet/pods/85a663b2-8dd3-4532-9269-a6048f484290/volumes" Oct 30 20:02:30.057086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:30.057044 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c0ab13-e06c-433e-995c-ba78139f9da1" path="/var/lib/kubelet/pods/b0c0ab13-e06c-433e-995c-ba78139f9da1/volumes" Oct 30 20:02:30.744286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:30.744227 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.435628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:31.435592 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.570566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:31.570529 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:31.724209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:31.724116 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.415113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:32.415038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.659560 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:32.654778 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.839141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:32.837176 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.326987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:33.326937 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.767220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:33.767184 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.154647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:34.154609 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.418145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:34.415263 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.629986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:34.629933 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:35.656113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:35.652580 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.583554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:37.583471 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.768083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:38.759989 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.069672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:39.069567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.447082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:39.446504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.402814 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.402773 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.496049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496010 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:02:40.496854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496830 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" containerName="payload-processing" Oct 30 20:02:40.496854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496851 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" containerName="payload-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496876 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b0c0ab13-e06c-433e-995c-ba78139f9da1" containerName="payload-pre-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496882 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c0ab13-e06c-433e-995c-ba78139f9da1" containerName="payload-pre-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496893 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496898 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496921 5733 cpu_manager.go:414] "containerMap: removing container" podUID="85a663b2-8dd3-4532-9269-a6048f484290" containerName="payload-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496926 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85a663b2-8dd3-4532-9269-a6048f484290" containerName="payload-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496942 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f43cb101-2fd1-4418-ac86-3920dc32477b" containerName="payload-pre-processing" Oct 30 20:02:40.497034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.496947 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43cb101-2fd1-4418-ac86-3920dc32477b" containerName="payload-pre-processing" Oct 30 20:02:40.497569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.497076 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad346e3-51bb-493b-a76f-71a08cca6a6b" containerName="payload-processing" Oct 30 20:02:40.497569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.497087 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c0ab13-e06c-433e-995c-ba78139f9da1" containerName="payload-pre-processing" Oct 30 20:02:40.497569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.497093 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd82080-4fc3-44b6-8554-f0548d18605c" containerName="istio-proxy" Oct 30 20:02:40.497569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.497102 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="85a663b2-8dd3-4532-9269-a6048f484290" containerName="payload-processing" Oct 30 20:02:40.497569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.497112 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43cb101-2fd1-4418-ac86-3920dc32477b" containerName="payload-pre-processing" Oct 30 20:02:40.506119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.506094 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.528793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.528762 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:02:40.565298 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.565264 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a7bb6c\"" Oct 30 20:02:40.566402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.566367 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.566617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.566596 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rlk\" (UniqueName: \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.602466 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.602437 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a7bb6c-dockercfg-5z5xs\"" Oct 30 20:02:40.671370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.668451 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.671370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.668709 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rlk\" (UniqueName: \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.677084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.674700 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.831221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.831186 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rlk\" (UniqueName: \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\") pod \"payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:40.878856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:40.877135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.125863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:41.125820 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:41.527649 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:02:41.527612 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod580aade9_3f41_41ff_a977_4a10a7814421.slice/crio-33729930ee1c1cdda6e6217164e804946c74a9778d4112a0ee5b2e07719e73ed WatchSource:0}: Error finding container 33729930ee1c1cdda6e6217164e804946c74a9778d4112a0ee5b2e07719e73ed: Status 404 returned error can't find the container with id 33729930ee1c1cdda6e6217164e804946c74a9778d4112a0ee5b2e07719e73ed Oct 30 20:02:41.529940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:41.529910 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:02:42.043041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:42.043002 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:42.275612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:42.275479 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" event={"ID":"580aade9-3f41-41ff-a977-4a10a7814421","Type":"ContainerStarted","Data":"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586"} Oct 30 20:02:42.275884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:42.275842 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:42.275994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:42.275898 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" event={"ID":"580aade9-3f41-41ff-a977-4a10a7814421","Type":"ContainerStarted","Data":"33729930ee1c1cdda6e6217164e804946c74a9778d4112a0ee5b2e07719e73ed"} Oct 30 20:02:42.336587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:42.336518 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" podStartSLOduration=2.336500252 podStartE2EDuration="2.336500252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:42.334500155 +0000 UTC m=+1922.402335726" watchObservedRunningTime="2026-10-30 20:02:42.336500252 +0000 UTC m=+1922.404335822" Oct 30 20:02:43.131570 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.131536 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:02:43.136017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.135993 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.156567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.156537 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4f244d-serving-cert\"" Oct 30 20:02:43.165840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.165808 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:02:43.242249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.242216 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.242448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.242282 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.242589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.242551 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.242701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.242681 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjgh\" (UniqueName: \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.323256 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.323217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.344239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.344199 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.344416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.344298 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjgh\" (UniqueName: \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.344475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.344425 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.344475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.344457 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.345871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.345844 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.347801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.347774 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.347962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.347938 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.360185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.360154 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjgh\" (UniqueName: \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\") pod \"maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.474262 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.474180 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:43.524004 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.523962 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:43.781890 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:02:43.781858 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b19494_6b48_41e6_956a_713a34b0dc10.slice/crio-2127bbaa810ed65d14f21f0ffcad86ebd044191916e25dab483af07f076e321a WatchSource:0}: Error finding container 2127bbaa810ed65d14f21f0ffcad86ebd044191916e25dab483af07f076e321a: Status 404 returned error can't find the container with id 2127bbaa810ed65d14f21f0ffcad86ebd044191916e25dab483af07f076e321a Oct 30 20:02:43.782752 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:43.782723 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:02:44.319778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:44.318742 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerStarted","Data":"2127bbaa810ed65d14f21f0ffcad86ebd044191916e25dab483af07f076e321a"} Oct 30 20:02:45.014341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:45.014301 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.331128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:45.330654 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerStarted","Data":"4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df"} Oct 30 20:02:45.331128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:45.330721 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:45.363287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:45.363238 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podStartSLOduration=1.8212463840000002 podStartE2EDuration="2.363223718s" totalImagesPullingTime="541.977334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:45.359200629 +0000 UTC m=+1925.427036207" watchObservedRunningTime="2026-10-30 20:02:45.363223718 +0000 UTC m=+1925.431059285" Oct 30 20:02:46.242667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:46.242621 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.503264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:46.503140 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.738595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:46.738554 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 20:02:46.738994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:46.738891 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" containerID="cri-o://826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d" gracePeriod=30 Oct 30 20:02:46.801154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:46.801045 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.683836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:47.682997 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:48.725989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:48.725805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.536532 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:49.536496 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.729887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:49.729852 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:50.930583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:50.930530 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.354955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:51.354927 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:02:51.620897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:51.620803 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.781980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:51.781955 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.988117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:51.986618 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.073495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.073084 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:02:52.135505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.135460 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.190217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190127 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190238 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190377 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190531 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:52.190649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190572 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190603 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190637 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190664 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:52.190797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190702 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190780 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.190934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190802 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:52.190934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.190874 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\") pod \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\" (UID: \"e5797ce7-d6ba-4ca6-9afd-4692c1333161\") " Oct 30 20:02:52.191152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.191120 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:52.191223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.191150 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:52.192681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.192595 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.192681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.192616 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.192681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.192629 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.192681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.192642 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.192681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.192655 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.193241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.193216 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:52.193680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.193658 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:52.193776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.193687 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "kube-api-access-4qjbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:52.193841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.193817 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy" pod "e5797ce7-d6ba-4ca6-9afd-4692c1333161" (UID: "e5797ce7-d6ba-4ca6-9afd-4692c1333161"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:52.293968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.293931 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.293968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.293962 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.293968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.293972 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qjbr\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-kube-api-access-4qjbr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.294239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.293984 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5797ce7-d6ba-4ca6-9afd-4692c1333161-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:52.419605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.419571 5733 generic.go:356] "Generic (PLEG): container finished" podID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerID="826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d" exitCode=0 Oct 30 20:02:52.419794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.419630 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" Oct 30 20:02:52.419794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.419660 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerDied","Data":"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d"} Oct 30 20:02:52.419794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.419725 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb" event={"ID":"e5797ce7-d6ba-4ca6-9afd-4692c1333161","Type":"ContainerDied","Data":"9bee9c18dab650ef30c1d449cbf4f4dc3af8b8025ba5d419bd8a35ee7680802d"} Oct 30 20:02:52.419794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.419750 5733 scope.go:122] "RemoveContainer" containerID="826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d" Oct 30 20:02:52.433227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.433205 5733 scope.go:122] "RemoveContainer" containerID="9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a" Oct 30 20:02:52.444917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.444897 5733 scope.go:122] "RemoveContainer" containerID="826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d" Oct 30 20:02:52.445243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:02:52.445223 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d\": container with ID starting with 826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d not found: ID does not exist" containerID="826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d" Oct 30 20:02:52.445338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.445251 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d"} err="failed to get container status \"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d\": rpc error: code = NotFound desc = could not find container \"826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d\": container with ID starting with 826bb0c1c86719670a445c0e94d6b101fe2a3a839ea8ec1e349af5e8014bea4d not found: ID does not exist" Oct 30 20:02:52.445338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.445268 5733 scope.go:122] "RemoveContainer" containerID="9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a" Oct 30 20:02:52.446121 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:02:52.445690 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a\": container with ID starting with 9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a not found: ID does not exist" containerID="9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a" Oct 30 20:02:52.446121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.445728 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a"} err="failed to get container status \"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a\": rpc error: code = NotFound desc = could not find container \"9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a\": container with ID starting with 9189255a516d6465eb860607583fa4142a73e764ed647bbf17ec4428a190695a not found: ID does not exist" Oct 30 20:02:52.452837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.452811 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 20:02:52.457759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:52.457727 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-35deec-openshift-default-7d8f8d655f-zdqvb"] Oct 30 20:02:53.047817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:53.047780 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.305643 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:53.305567 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:02:54.054966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:54.054936 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" path="/var/lib/kubelet/pods/e5797ce7-d6ba-4ca6-9afd-4692c1333161/volumes" Oct 30 20:02:54.130155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:54.130118 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.132778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:55.132736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.011078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:56.011010 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.557033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:56.556995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.815534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:56.815419 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.093233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:57.093153 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.359976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:57.359945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.624920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:57.624786 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.786028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:57.785992 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.137415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:58.137383 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.645648 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:58.645610 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.910824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:02:58.910732 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:59.379324 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:02:59.379292 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:03:00.015103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:00.015045 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:00.839722 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:00.838629 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.804979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:01.804939 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.272553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:02.272519 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.472160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:02.472126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.954589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:02.954549 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.567422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:03.567390 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.804238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:03.804201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.355926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.355890 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:03:04.357336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.357254 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.357796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.357590 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.358939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.358917 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.359153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.359137 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.359796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.359775 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.359936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.359925 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5797ce7-d6ba-4ca6-9afd-4692c1333161" containerName="istio-proxy" Oct 30 20:03:04.365266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.365246 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.367259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.367235 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:03:04.388366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.388334 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a7bb6c-serving-cert\"" Oct 30 20:03:04.471645 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.471601 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.510640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.508370 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.510640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.508447 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2vm\" (UniqueName: \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.510640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.508674 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.510640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.508796 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.610635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.610526 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.610823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.610665 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.610823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.610805 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.610951 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.610860 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2vm\" (UniqueName: \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.611503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.611474 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.613808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.613784 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.613936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.613870 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.628828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.628736 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2vm\" (UniqueName: \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\") pod \"maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.682405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.682374 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:04.705287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.705259 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.986155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.985563 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:03:04.994010 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:04.993078 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:03:05.596487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:05.596450 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerStarted","Data":"3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905"} Oct 30 20:03:05.596863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:05.596551 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerStarted","Data":"0f5028e2c38b8218ca934754132cac5f323b64978c15727a8df381d5058310a2"} Oct 30 20:03:05.596863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:05.596591 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:05.638203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:05.638151 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podStartSLOduration=1.175608399 podStartE2EDuration="1.638134943s" totalImagesPullingTime="462.526544ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:05.632870119 +0000 UTC m=+1945.700705687" watchObservedRunningTime="2026-10-30 20:03:05.638134943 +0000 UTC m=+1945.705970511" Oct 30 20:03:05.788903 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:05.788794 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.537121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:06.534351 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.723022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:06.722976 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.864994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:06.864957 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:03:06.865238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:06.865213 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" containerID="cri-o://973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311" gracePeriod=30 Oct 30 20:03:06.993636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:06.993598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:08.250083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:08.243608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.324631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:09.324370 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.088206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.088167 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.542018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.540121 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.810593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.810518 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.954932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.954896 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:03:10.981361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.981322 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:03:10.981559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.981393 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:10.990449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:10.990422 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-77b4df-openshift-default-dockercfg-47njs\"" Oct 30 20:03:11.053435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.053388 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.069193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069108 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.069342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069271 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.069342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069311 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7gz8\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.069521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069451 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.069654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069634 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.069868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.069844 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.070033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.070014 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.070423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.070398 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.070517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.070476 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.186905 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.186960 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7gz8\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187106 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187204 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187292 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187342 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187446 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187512 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.187547 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.188022 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.192778 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.193082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.193018 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.219078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.218173 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.219078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.218440 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.219078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.218905 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.219455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.219440 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.247097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.246114 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.251081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.248181 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7gz8\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\") pod \"e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.265166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.265079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.302079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.299597 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:11.616140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.612759 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:03:11.688858 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:03:11.688817 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaffa7e70_81df_440e_a795_9e0b5ee3ae27.slice/crio-be24f0b9d2dc4d5a8034c95087058948a6b0f5c90c69470de644dc9fc68da5b5 WatchSource:0}: Error finding container be24f0b9d2dc4d5a8034c95087058948a6b0f5c90c69470de644dc9fc68da5b5: Status 404 returned error can't find the container with id be24f0b9d2dc4d5a8034c95087058948a6b0f5c90c69470de644dc9fc68da5b5 Oct 30 20:03:11.690413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.690385 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:03:11.692743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.692703 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.692863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.692792 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.692863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.692838 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:11.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:11.955563 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.179647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.179607 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:03:12.232848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.232800 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.232885 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.232953 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.232986 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.233031 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.233191 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.233238 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.233368 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.233695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.233485 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\") pod \"4f0f0fc7-8645-4720-86f7-f628266648c8\" (UID: \"4f0f0fc7-8645-4720-86f7-f628266648c8\") " Oct 30 20:03:12.234190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.234163 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:12.234862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.234831 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.235194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.235166 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:12.235291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.235188 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:12.235291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.235259 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:12.235448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.235424 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:12.245165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.239789 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:12.245165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.242919 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:12.245165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.245554 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:12.259285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.256434 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6" pod "4f0f0fc7-8645-4720-86f7-f628266648c8" (UID: "4f0f0fc7-8645-4720-86f7-f628266648c8"). InnerVolumeSpecName "kube-api-access-gp6h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337286 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337322 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337336 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f0f0fc7-8645-4720-86f7-f628266648c8-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337358 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f0f0fc7-8645-4720-86f7-f628266648c8-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337374 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337388 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp6h6\" (UniqueName: \"kubernetes.io/projected/4f0f0fc7-8645-4720-86f7-f628266648c8-kube-api-access-gp6h6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337404 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.337453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.337417 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f0f0fc7-8645-4720-86f7-f628266648c8-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:12.667376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.667344 5733 generic.go:356] "Generic (PLEG): container finished" podID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerID="973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311" exitCode=0 Oct 30 20:03:12.667836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.667393 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" Oct 30 20:03:12.667836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.667425 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" event={"ID":"4f0f0fc7-8645-4720-86f7-f628266648c8","Type":"ContainerDied","Data":"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311"} Oct 30 20:03:12.667836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.667479 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6" event={"ID":"4f0f0fc7-8645-4720-86f7-f628266648c8","Type":"ContainerDied","Data":"db5fd12a40bfc0e6c7eab65500f97c0b77b5433e647ae1b4e92d37bb1f1e162d"} Oct 30 20:03:12.667836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.667503 5733 scope.go:122] "RemoveContainer" containerID="973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311" Oct 30 20:03:12.669647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.669615 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerStarted","Data":"06977bec715dad5676938c01cef6f28c9d4b1a9d746da24f7e03ad44cf9754a4"} Oct 30 20:03:12.669748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.669671 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerStarted","Data":"be24f0b9d2dc4d5a8034c95087058948a6b0f5c90c69470de644dc9fc68da5b5"} Oct 30 20:03:12.679102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.679083 5733 scope.go:122] "RemoveContainer" containerID="973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311" Oct 30 20:03:12.679379 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:03:12.679360 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311\": container with ID starting with 973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311 not found: ID does not exist" containerID="973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311" Oct 30 20:03:12.679443 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.679386 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311"} err="failed to get container status \"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311\": rpc error: code = NotFound desc = could not find container \"973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311\": container with ID starting with 973a96df065cab10f7e8cc23788a5cb71928cd3e35c678061bcc2565bad48311 not found: ID does not exist" Oct 30 20:03:12.703854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.703801 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podStartSLOduration=2.703788276 podStartE2EDuration="2.703788276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:12.699603733 +0000 UTC m=+1952.767439302" watchObservedRunningTime="2026-10-30 20:03:12.703788276 +0000 UTC m=+1952.771623843" Oct 30 20:03:12.724445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.724412 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:03:12.728989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.728959 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c50e1e-openshift-default-88df8fc95-mc5f6"] Oct 30 20:03:12.801743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:12.801716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:13.300181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:13.300151 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:13.306603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:13.306576 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:13.725549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:13.725514 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:13.728641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:13.728614 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:03:14.054607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:14.054509 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" path="/var/lib/kubelet/pods/4f0f0fc7-8645-4720-86f7-f628266648c8/volumes" Oct 30 20:03:14.161070 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:14.161019 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.227394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:15.227353 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.397085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:15.394132 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.810470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:15.810423 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.326939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.326896 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.490414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.490380 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.661201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.661167 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.806595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.806558 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:03:16.808201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.808162 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" Oct 30 20:03:16.808374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.808360 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" Oct 30 20:03:16.911982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.911898 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f0f0fc7-8645-4720-86f7-f628266648c8" containerName="istio-proxy" Oct 30 20:03:16.923178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.923148 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:16.923873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.923844 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:03:16.931530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.931324 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-77b4df\"" Oct 30 20:03:16.932157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:16.931373 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-77b4df-dockercfg-zvrq6\"" Oct 30 20:03:17.007910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.007872 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.008089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.008027 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rl2n\" (UniqueName: \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.025107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.022767 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:17.110339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.110290 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.110559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.110510 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rl2n\" (UniqueName: \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.119863 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.119826 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.134737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.134640 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rl2n\" (UniqueName: \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\") pod \"payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.246481 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.246389 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.596336 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:03:17.596305 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05d05bb_294b_4d00_a60f_98fd0d84bfdb.slice/crio-642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f WatchSource:0}: Error finding container 642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f: Status 404 returned error can't find the container with id 642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f Oct 30 20:03:17.596832 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.596768 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:03:17.618975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.618942 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:17.783965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.778612 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:17.891084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.887551 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" event={"ID":"d05d05bb-294b-4d00-a60f-98fd0d84bfdb","Type":"ContainerStarted","Data":"cde33313a9ac503b0a84a13efd688ee7ef3121b45f8c55a793d9e31695da2c7c"} Oct 30 20:03:17.891084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.887622 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" event={"ID":"d05d05bb-294b-4d00-a60f-98fd0d84bfdb","Type":"ContainerStarted","Data":"642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f"} Oct 30 20:03:17.891084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.889021 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:17.963041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:17.961422 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podStartSLOduration=1.961400179 podStartE2EDuration="1.961400179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:17.915431485 +0000 UTC m=+1957.983267070" watchObservedRunningTime="2026-10-30 20:03:17.961400179 +0000 UTC m=+1958.029235748" Oct 30 20:03:18.516885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:18.516842 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.080529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:20.079265 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:21.198107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:21.198038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:21.390916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:21.390879 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:21.769126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:21.766382 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.457905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.457862 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.498429 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.498394 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:03:22.503758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.503233 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.538886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.536720 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82885f-dockercfg-jddkg\"" Oct 30 20:03:22.587086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.566775 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82885f\"" Oct 30 20:03:22.597081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.595175 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.597081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.595509 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v82q6\" (UniqueName: \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.664088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.661706 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:22.700873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.698712 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v82q6\" (UniqueName: \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.700873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.698819 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.700873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.700024 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.700873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.700313 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:03:22.883971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.883942 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v82q6\" (UniqueName: \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\") pod \"payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:22.943133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.942343 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:03:22.969071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.969020 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:03:22.973775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.973701 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:22.976551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.976522 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mffh\" (UniqueName: \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:22.976686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:22.976560 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.066123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.062826 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:23.078622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.078587 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mffh\" (UniqueName: \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.078811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.078800 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.079657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.079637 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.092672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.092645 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mffh\" (UniqueName: \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\") pod \"payload-processing-e2e-inf-b-82885f-55777b478d-khbl8\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.193358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.193268 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:23.305939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.305301 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:23.586855 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.585281 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:03:23.590190 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:03:23.590156 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9306a89_0fc2_44cd_9aa2_96f5ec782c02.slice/crio-addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d WatchSource:0}: Error finding container addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d: Status 404 returned error can't find the container with id addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d Oct 30 20:03:23.706984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:23.706957 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:03:24.006489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.006405 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" event={"ID":"382161a6-f53d-4795-b7c4-005dd3e1ca94","Type":"ContainerStarted","Data":"4dddfd0903c2e64409656e5498618384143b72a74c38171ae766e6b7da07c1ce"} Oct 30 20:03:24.006489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.006475 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:24.006667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.006501 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" event={"ID":"382161a6-f53d-4795-b7c4-005dd3e1ca94","Type":"ContainerStarted","Data":"8a331bea8150ae62ef394e67a2e3be43c3e98cbfd635147fe91a5778589a941d"} Oct 30 20:03:24.008050 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.008024 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" event={"ID":"c9306a89-0fc2-44cd-9aa2-96f5ec782c02","Type":"ContainerStarted","Data":"78f444055ce64fe1646d83493086c6f1422e2aba200d308410827f191a4324ff"} Oct 30 20:03:24.008194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.008078 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" event={"ID":"c9306a89-0fc2-44cd-9aa2-96f5ec782c02","Type":"ContainerStarted","Data":"addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d"} Oct 30 20:03:24.008194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.008108 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:24.041581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.041524 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podStartSLOduration=2.041507185 podStartE2EDuration="2.041507185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:24.040621269 +0000 UTC m=+1964.108456838" watchObservedRunningTime="2026-10-30 20:03:24.041507185 +0000 UTC m=+1964.109342753" Oct 30 20:03:24.072714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.072664 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podStartSLOduration=2.072651005 podStartE2EDuration="2.072651005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:24.066921811 +0000 UTC m=+1964.134757381" watchObservedRunningTime="2026-10-30 20:03:24.072651005 +0000 UTC m=+1964.140486573" Oct 30 20:03:24.262462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:24.262360 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:25.375406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:25.375372 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:25.526081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:25.525811 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:25.834142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:25.833639 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.624131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:26.617727 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:27.526651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:27.526561 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.400874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:28.400830 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.967582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:28.967546 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:03:28.968087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:28.968038 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" podUID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" containerName="maas-api" containerID="cri-o://c60feac2558d6404b9a04e35f405e39089ee73ce1f3c193225d68cde991a26d5" gracePeriod=30 Oct 30 20:03:29.105602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.105566 5733 generic.go:356] "Generic (PLEG): container finished" podID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" containerID="c60feac2558d6404b9a04e35f405e39089ee73ce1f3c193225d68cde991a26d5" exitCode=0 Oct 30 20:03:29.105763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.105693 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" event={"ID":"7854f0d2-ea62-4808-bc07-5bfdd05b55f2","Type":"ContainerDied","Data":"c60feac2558d6404b9a04e35f405e39089ee73ce1f3c193225d68cde991a26d5"} Oct 30 20:03:29.108538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.108513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.591579 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:03:29.591532 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:03:29.616454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.616426 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:03:29.684239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.684208 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\") pod \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " Oct 30 20:03:29.684408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.684278 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\" (UniqueName: \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\") pod \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " Oct 30 20:03:29.684600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.684575 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\") pod \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " Oct 30 20:03:29.684725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.684626 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca" pod "7854f0d2-ea62-4808-bc07-5bfdd05b55f2" (UID: "7854f0d2-ea62-4808-bc07-5bfdd05b55f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:29.684725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.684644 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\") pod \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\" (UID: \"7854f0d2-ea62-4808-bc07-5bfdd05b55f2\") " Oct 30 20:03:29.685524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.685495 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:29.686809 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.686781 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm" pod "7854f0d2-ea62-4808-bc07-5bfdd05b55f2" (UID: "7854f0d2-ea62-4808-bc07-5bfdd05b55f2"). InnerVolumeSpecName "kube-api-access-wmhhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:29.686897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.686813 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs" pod "7854f0d2-ea62-4808-bc07-5bfdd05b55f2" (UID: "7854f0d2-ea62-4808-bc07-5bfdd05b55f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:29.686942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.686899 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls" pod "7854f0d2-ea62-4808-bc07-5bfdd05b55f2" (UID: "7854f0d2-ea62-4808-bc07-5bfdd05b55f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:29.737714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.737682 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.787853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.787819 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhhm\" (UniqueName: \"kubernetes.io/projected/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-kube-api-access-wmhhm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:29.787853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.787847 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:29.787853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.787858 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7854f0d2-ea62-4808-bc07-5bfdd05b55f2-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:29.885422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.885378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.916090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:29.914855 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:03:30.028620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.028037 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.120079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.118849 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" event={"ID":"7854f0d2-ea62-4808-bc07-5bfdd05b55f2","Type":"ContainerDied","Data":"32b150792356f6886360daea565b2f5cd415fdac52dc8270e1b788f4515257a8"} Oct 30 20:03:30.120079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.118921 5733 scope.go:122] "RemoveContainer" containerID="c60feac2558d6404b9a04e35f405e39089ee73ce1f3c193225d68cde991a26d5" Oct 30 20:03:30.120079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.119108 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv" Oct 30 20:03:30.149129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.149042 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:03:30.156208 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.156182 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-ed8b50-57474df78d-k8bkv"] Oct 30 20:03:30.393506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.393473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.787087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:30.777692 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:31.622449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:31.622325 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:59110->10.133.0.157:15021: read: connection reset by peer" start-of-body= Oct 30 20:03:31.622449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:31.622415 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": read tcp 10.133.0.2:59110->10.133.0.157:15021: read: connection reset by peer" Oct 30 20:03:31.694943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:31.692219 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.054103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.053997 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" path="/var/lib/kubelet/pods/7854f0d2-ea62-4808-bc07-5bfdd05b55f2/volumes" Oct 30 20:03:32.140697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.140663 5733 generic.go:356] "Generic (PLEG): container finished" podID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerID="6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3" exitCode=0 Oct 30 20:03:32.140866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.140732 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerDied","Data":"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3"} Oct 30 20:03:32.141302 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.141283 5733 scope.go:122] "RemoveContainer" containerID="6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3" Oct 30 20:03:32.143272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.143240 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:32.143401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.143325 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:32.143401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.143374 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:32.895538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:32.894081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.172202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:33.172101 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerStarted","Data":"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14"} Oct 30 20:03:33.891221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:33.891080 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.022083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:34.016424 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:03:34.022083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:34.016588 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:03:34.024177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:34.024130 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:03:34.829804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:34.829767 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.010425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.010386 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:03:35.010721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.010687 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:03:35.051696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.051667 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:03:35.051870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.051729 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:03:35.091106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.090992 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.426553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.426501 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.775248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.775178 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:03:35.776228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.776208 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" containerName="maas-api" Oct 30 20:03:35.776350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.776341 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" containerName="maas-api" Oct 30 20:03:35.776580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.776569 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7854f0d2-ea62-4808-bc07-5bfdd05b55f2" containerName="maas-api" Oct 30 20:03:35.779613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.779592 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.794998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.794973 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:03:35.795780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.795757 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d7a733-openshift-default-dockercfg-dwpvw\"" Oct 30 20:03:35.850630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850597 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850641 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hzt\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850674 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850731 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850761 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850789 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.850836 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.851087 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.851167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.851144 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:35.924104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.923651 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:35.990117 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.001933 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002041 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002241 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002274 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hzt\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002312 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002391 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002424 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002453 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.023095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.002510 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.030418 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.030479 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.031186 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.031864 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.032129 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.043482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.032394 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.114164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.081866 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.118123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.114566 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hzt\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.209145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.125396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.209145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.148688 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\") pod \"e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.209145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.168707 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.401835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.401803 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:36.550660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.522499 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.990837 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:03:36.990807 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78a1582_b7a4_4742_a25a_94209cae3c42.slice/crio-d2988e71f2a7f997235f4863d576425f1bff04c26f6cee71c3fa6b765c8c9ea5 WatchSource:0}: Error finding container d2988e71f2a7f997235f4863d576425f1bff04c26f6cee71c3fa6b765c8c9ea5: Status 404 returned error can't find the container with id d2988e71f2a7f997235f4863d576425f1bff04c26f6cee71c3fa6b765c8c9ea5 Oct 30 20:03:36.993142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.993098 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:03:36.994584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.994552 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:36.994637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.994620 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:36.994688 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:36.994675 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:03:37.044092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.042325 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:03:37.044092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.042395 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:03:37.291763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.291664 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerStarted","Data":"5952e30aa9cfb6a17ce055ac00ba93d8c9904548c52d436e8214f25ec120003c"} Oct 30 20:03:37.291763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.291732 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerStarted","Data":"d2988e71f2a7f997235f4863d576425f1bff04c26f6cee71c3fa6b765c8c9ea5"} Oct 30 20:03:37.361597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.361524 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podStartSLOduration=2.361502252 podStartE2EDuration="2.361502252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:37.329489292 +0000 UTC m=+1977.397324864" watchObservedRunningTime="2026-10-30 20:03:37.361502252 +0000 UTC m=+1977.429337821" Oct 30 20:03:37.402284 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.402255 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:37.662279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.662236 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:03:37.671795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.671767 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.672169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.672143 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:03:37.687344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.687319 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-77b4df-serving-cert\"" Oct 30 20:03:37.766464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.766420 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.766668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.766544 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.766762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.766738 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5cgd\" (UniqueName: \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.766819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.766791 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.768373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.768350 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.885080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.884804 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5cgd\" (UniqueName: \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.885080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.884895 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.885363 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.885118 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.885363 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.885226 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.886369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.886106 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.890098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.889362 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.890098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.889687 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.941140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.933346 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5cgd\" (UniqueName: \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\") pod \"maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:37.991180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:37.991148 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:38.039082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:38.024445 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 20:03:38.039082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:38.024525 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 20:03:38.146080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:38.133604 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.345095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:38.344858 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:03:38.345988 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:03:38.345955 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c8e7e1_6494_49a8_b61c_bb3098dcccde.slice/crio-fcdf1350e6b3e6328fabe43827bcbeb557992a173c5b1122a483ce8900906fdc WatchSource:0}: Error finding container fcdf1350e6b3e6328fabe43827bcbeb557992a173c5b1122a483ce8900906fdc: Status 404 returned error can't find the container with id fcdf1350e6b3e6328fabe43827bcbeb557992a173c5b1122a483ce8900906fdc Oct 30 20:03:38.409148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:38.409097 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:39.015725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.015693 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:03:39.016247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.016083 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:03:39.016972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.016941 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:03:39.311066 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.311020 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerStarted","Data":"8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d"} Oct 30 20:03:39.311238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.311095 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerStarted","Data":"fcdf1350e6b3e6328fabe43827bcbeb557992a173c5b1122a483ce8900906fdc"} Oct 30 20:03:39.311238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.311127 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:39.311853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.311824 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:39.312729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.312704 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:03:39.332128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.327302 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.352234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.352169 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podStartSLOduration=1.862254508 podStartE2EDuration="2.352148096s" totalImagesPullingTime="489.893588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:39.35134361 +0000 UTC m=+1979.419179181" watchObservedRunningTime="2026-10-30 20:03:39.352148096 +0000 UTC m=+1979.419983664" Oct 30 20:03:39.501454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.501424 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.783425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.783394 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.984907 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:39.984871 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.412919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:40.412883 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.660830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:40.660788 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.903749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:40.903715 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.313136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:41.312651 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.500627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:41.500555 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.715659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:41.715123 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.089720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:42.089630 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.340236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:42.338329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.120226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:44.120191 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.338604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:45.338575 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:03:45.440474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:45.440439 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.867471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:45.867435 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.202106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:46.201946 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.580856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:46.580779 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:03:46.581616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:46.581586 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" podUID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" containerName="maas-api" containerID="cri-o://68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9" gracePeriod=30 Oct 30 20:03:46.720196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:46.720168 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:46.954286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:46.944631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.016646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.015285 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:03:47.153738 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.152170 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\") pod \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " Oct 30 20:03:47.153738 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.152273 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") pod \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " Oct 30 20:03:47.153738 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.152344 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\") pod \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " Oct 30 20:03:47.153738 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.152541 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\" (UniqueName: \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\") pod \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\" (UID: \"b46ecfd4-5a1b-4270-8712-c3ff7be83f03\") " Oct 30 20:03:47.161565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.160937 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca" pod "b46ecfd4-5a1b-4270-8712-c3ff7be83f03" (UID: "b46ecfd4-5a1b-4270-8712-c3ff7be83f03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:47.161565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.161006 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz" pod "b46ecfd4-5a1b-4270-8712-c3ff7be83f03" (UID: "b46ecfd4-5a1b-4270-8712-c3ff7be83f03"). InnerVolumeSpecName "kube-api-access-chzlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:47.174782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.174716 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs" pod "b46ecfd4-5a1b-4270-8712-c3ff7be83f03" (UID: "b46ecfd4-5a1b-4270-8712-c3ff7be83f03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:47.175092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.175049 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls" pod "b46ecfd4-5a1b-4270-8712-c3ff7be83f03" (UID: "b46ecfd4-5a1b-4270-8712-c3ff7be83f03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:47.255068 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.255018 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:47.255274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.255261 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzlz\" (UniqueName: \"kubernetes.io/projected/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-kube-api-access-chzlz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:47.255341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.255330 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:47.255436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.255423 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46ecfd4-5a1b-4270-8712-c3ff7be83f03-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:47.493159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.493026 5733 generic.go:356] "Generic (PLEG): container finished" podID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" containerID="68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9" exitCode=0 Oct 30 20:03:47.493459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.493435 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" Oct 30 20:03:47.493707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.493426 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" event={"ID":"b46ecfd4-5a1b-4270-8712-c3ff7be83f03","Type":"ContainerDied","Data":"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9"} Oct 30 20:03:47.493886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.493865 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz" event={"ID":"b46ecfd4-5a1b-4270-8712-c3ff7be83f03","Type":"ContainerDied","Data":"609ab4ac9fc902505458a3ffda24d6584a0e1cb6fabeda731af9681e182cc25f"} Oct 30 20:03:47.494013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.493989 5733 scope.go:122] "RemoveContainer" containerID="68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9" Oct 30 20:03:47.537296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.537221 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.592155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.592122 5733 scope.go:122] "RemoveContainer" containerID="68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9" Oct 30 20:03:47.611453 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:03:47.611099 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9\": container with ID starting with 68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9 not found: ID does not exist" containerID="68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9" Oct 30 20:03:47.611453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.611147 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9"} err="failed to get container status \"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9\": rpc error: code = NotFound desc = could not find container \"68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9\": container with ID starting with 68109e62acef2968f008ef09dec9203f1a77ea7ab939de8256361b9c0cd734a9 not found: ID does not exist" Oct 30 20:03:47.631212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.631180 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:03:47.653104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.647524 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-96ec93-5d6d47bf89-s5pcz"] Oct 30 20:03:47.705415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.705375 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:03:47.706883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.706848 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" containerName="maas-api" Oct 30 20:03:47.706883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.706875 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" containerName="maas-api" Oct 30 20:03:47.707269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.707253 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" containerName="maas-api" Oct 30 20:03:47.713689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.713656 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.728408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.728365 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-925e04\"" Oct 30 20:03:47.734823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.734796 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:03:47.744346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.744224 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:03:47.749806 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.749712 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:47.777400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.774460 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-925e04-dockercfg-tstq2\"" Oct 30 20:03:47.789071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.779659 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:03:47.806028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.805999 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.870232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.870201 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lswj8\" (UniqueName: \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.870445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.870243 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.870445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.870406 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:47.870573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.870503 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pjzn\" (UniqueName: \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.984211 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lswj8\" (UniqueName: \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.984265 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.984456 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.984552 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pjzn\" (UniqueName: \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.986085 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:47.990079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:47.986748 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:48.070239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.050325 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.095114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.094776 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pjzn\" (UniqueName: \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\") pod \"payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:48.104038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.103775 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:48.155200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.148831 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lswj8\" (UniqueName: \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\") pod \"payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:48.156162 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.156117 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46ecfd4-5a1b-4270-8712-c3ff7be83f03" path="/var/lib/kubelet/pods/b46ecfd4-5a1b-4270-8712-c3ff7be83f03/volumes" Oct 30 20:03:48.346440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.346319 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:48.504078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.500848 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.688130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.647562 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:03:48.862079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.846767 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.880723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.880459 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:03:48.974121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.974089 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:03:48.981567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.981544 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:48.993660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.993634 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:03:48.994823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.994803 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f5eee\"" Oct 30 20:03:48.997737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:48.997701 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f5eee-dockercfg-76ns8\"" Oct 30 20:03:49.066657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.064879 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcbpx\" (UniqueName: \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.066657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.065036 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.093346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.082951 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.195510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.193847 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcbpx\" (UniqueName: \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.195510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.193906 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.195510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.195189 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.335193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.334763 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcbpx\" (UniqueName: \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\") pod \"payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.553695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.553652 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" event={"ID":"cb994d3f-ac88-4fff-b9ee-400f5556cafb","Type":"ContainerStarted","Data":"02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9"} Oct 30 20:03:49.553901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.553724 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" event={"ID":"cb994d3f-ac88-4fff-b9ee-400f5556cafb","Type":"ContainerStarted","Data":"52a1c191c33d86cc4d2fdd81fb1affb6e062fa174e1c776b21726dd6becaf840"} Oct 30 20:03:49.553901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.553788 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:03:49.555861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.555831 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" event={"ID":"3433ab80-4a84-4013-80d2-7c7555c470d3","Type":"ContainerStarted","Data":"69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98"} Oct 30 20:03:49.555986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.555899 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" event={"ID":"3433ab80-4a84-4013-80d2-7c7555c470d3","Type":"ContainerStarted","Data":"0dfe2d739c1c44b8936879da9e7ca75d2c4b2e76214f576ca277e6a305cab2c8"} Oct 30 20:03:49.555986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.555941 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:03:49.583115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.583049 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podStartSLOduration=2.583036856 podStartE2EDuration="2.583036856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:49.580189883 +0000 UTC m=+1989.648025450" watchObservedRunningTime="2026-10-30 20:03:49.583036856 +0000 UTC m=+1989.650872424" Oct 30 20:03:49.600803 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.600780 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:49.608314 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.608266 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podStartSLOduration=2.608254393 podStartE2EDuration="2.608254393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:49.603775905 +0000 UTC m=+1989.671611475" watchObservedRunningTime="2026-10-30 20:03:49.608254393 +0000 UTC m=+1989.676089961" Oct 30 20:03:49.895114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.895023 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.917174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:49.916591 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:03:50.569407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:50.569367 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" event={"ID":"058c7110-65ab-4311-9688-b389e30c5f2a","Type":"ContainerStarted","Data":"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502"} Oct 30 20:03:50.569627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:50.569431 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" event={"ID":"058c7110-65ab-4311-9688-b389e30c5f2a","Type":"ContainerStarted","Data":"51a10eaddd3cbf5a769b7e2dbe1f2f1dda892ba1fa9eaa1b5bdcf04ccaaa1287"} Oct 30 20:03:50.569800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:50.569773 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:03:50.599763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:50.599715 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podStartSLOduration=2.599700614 podStartE2EDuration="2.599700614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:50.593860469 +0000 UTC m=+1990.661696041" watchObservedRunningTime="2026-10-30 20:03:50.599700614 +0000 UTC m=+1990.667536178" Oct 30 20:03:51.121449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:51.121250 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.096831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:52.090373 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.283426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:52.283393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.451760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:52.451672 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.028079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:53.021874 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.812716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:53.812583 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:54.948106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:54.946157 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:55.588664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:55.588626 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:55.732780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:55.732749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.018544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:56.018503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.172081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:56.171915 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.043963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:57.043923 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.293373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:57.293331 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.883971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:57.882165 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.029307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:58.029245 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.586603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:58.586570 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.758278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:58.758232 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.379252 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:03:59.379217 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:03:59.573568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:03:59.573473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:00.575509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:00.575475 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:04:00.576669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:00.576641 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:04:00.630753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:00.630690 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.275875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:01.275839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.437569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:01.437513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.588940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:01.588792 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:04:01.604753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:01.604696 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.129861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.129829 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.812332 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.812288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.919978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.919944 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:04:02.924963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.924934 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:02.936447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.936418 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:04:02.974127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.966611 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7f7d21\"" Oct 30 20:04:02.974127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:02.969423 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7f7d21-dockercfg-9v9wb\"" Oct 30 20:04:03.000142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.000041 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.038381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.038215 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.039135 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.038916 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtj4t\" (UniqueName: \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.148599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.148554 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtj4t\" (UniqueName: \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.156450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.156409 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.174126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.157792 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.221080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.202108 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtj4t\" (UniqueName: \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\") pod \"payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.256110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.256074 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:03.810212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.807740 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:04:03.845081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:03.843665 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.054084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:05.037085 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" event={"ID":"8ed984b5-27dc-47f6-b3a3-0be88a09763e","Type":"ContainerStarted","Data":"3714c0014297877970558d020c838c846f37d5b13e99c3bf65564e5837da6db9"} Oct 30 20:04:05.054084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:05.037157 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" event={"ID":"8ed984b5-27dc-47f6-b3a3-0be88a09763e","Type":"ContainerStarted","Data":"c6bd2fc509ab2dd73a0784824be54f1def5924c2a1b9192cc39704525e85f96c"} Oct 30 20:04:05.054084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:05.038446 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:05.080482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:05.080421 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" podStartSLOduration=3.080403317 podStartE2EDuration="3.080403317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:05.076098519 +0000 UTC m=+2005.143934090" watchObservedRunningTime="2026-10-30 20:04:05.080403317 +0000 UTC m=+2005.148238889" Oct 30 20:04:05.752737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:05.752669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.137080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.136354 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.267080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.254143 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.294752 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.294702 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.328138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.328095 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.492246 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.492146 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.505393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.505346 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.540086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.536131 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.818142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.818030 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.835809 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.834846 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:06.862380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:06.861375 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.029683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.029616 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.144929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.144874 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.187082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.158006 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:04:07.341579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.341532 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 20:04:07.359269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.359220 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.381091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.376574 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 20:04:07.381091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.376980 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.381091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.377926 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:04:07.381091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.378009 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:04:07.441268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.441169 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:07.441572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.441535 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.540079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.538906 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.560088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.540944 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:04:07.560698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.560659 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:04:07.562817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.562775 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.563509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.563486 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:04:07.563759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.563743 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:04:07.580190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.566576 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:07.791970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.791840 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:04:07.860679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.860647 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:04:07.860877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.860708 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:04:07.890088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.889271 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:04:07.932836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.932809 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:04:07.962945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:07.962918 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:04:08.866113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:08.866040 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.108690 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:10.108661 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.268507 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:10.268466 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.433598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:10.433506 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:11.490815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:11.490777 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:11.957652 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:11.957611 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.202715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:13.202678 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.225130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:14.219796 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.767243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:14.767198 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:04:14.767426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:14.767283 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:04:15.124015 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.123965 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.281216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.275526 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.606845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.606813 5733 generic.go:356] "Generic (PLEG): container finished" podID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerID="06977bec715dad5676938c01cef6f28c9d4b1a9d746da24f7e03ad44cf9754a4" exitCode=0 Oct 30 20:04:15.607009 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.606885 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerDied","Data":"06977bec715dad5676938c01cef6f28c9d4b1a9d746da24f7e03ad44cf9754a4"} Oct 30 20:04:15.607400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.607382 5733 scope.go:122] "RemoveContainer" containerID="06977bec715dad5676938c01cef6f28c9d4b1a9d746da24f7e03ad44cf9754a4" Oct 30 20:04:15.609266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.609238 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:15.609350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.609307 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:15.609350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:15.609339 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:16.400972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:16.400922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:16.620150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:16.620110 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerStarted","Data":"6e82609386271287934d9c795b6892c0b19f762044228933d9eb67c06ed03fb4"} Oct 30 20:04:17.301137 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.301104 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:04:17.301420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.301390 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:17.301470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.301452 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:17.466099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.464673 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:04:17.596071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.595415 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:17.781128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:17.775167 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.301096 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:18.300643 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:18.301096 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:18.300739 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:19.002321 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:19.002278 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.300540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:19.300463 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:19.300540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:19.300524 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:20.308469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:20.307732 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:20.308469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:20.307790 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:20.309511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:20.309480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:21.301171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:21.301133 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:21.301386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:21.301234 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:21.446939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:21.446904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.086013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:22.085977 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.301392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:22.301344 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 20:04:22.301630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:22.301412 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 20:04:22.404840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:22.404805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.321329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:23.321299 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:04:23.328346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:23.328183 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:04:23.346366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:23.346330 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:04:23.368728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:23.368693 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:24.505573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:24.505524 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:25.414379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:25.413881 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:26.268485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:26.268425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:26.962020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:26.961981 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:27.294802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:27.292712 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:27.771238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:27.771205 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 20:04:27.771428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:27.771402 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" containerID="cri-o://1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9" gracePeriod=30 Oct 30 20:04:27.953937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:27.953903 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 20:04:27.954155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:27.954130 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" containerID="cri-o://a1dc7fe9a61003310f76426a30fb6ae8c9d5af188a4d9d6eb9031e015f8badbc" gracePeriod=30 Oct 30 20:04:28.642453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:28.642385 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.379145 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:04:29.379104 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:04:29.527210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:29.527121 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.563116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:30.562689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.506867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:31.506816 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.671079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:31.667825 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.624585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:32.624541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:33.114882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.114608 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 20:04:33.116636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.116408 5733 generic.go:356] "Generic (PLEG): container finished" podID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerID="1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9" exitCode=0 Oct 30 20:04:33.116636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.116488 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" event={"ID":"81e3bbd1-0da1-4638-aaec-8bcf01b332a1","Type":"ContainerDied","Data":"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9"} Oct 30 20:04:33.116636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.116564 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" event={"ID":"81e3bbd1-0da1-4638-aaec-8bcf01b332a1","Type":"ContainerDied","Data":"a89d0d7c6c91a6d9620dcf11adc0a064edcaa30c9c2932b14b58439d91021330"} Oct 30 20:04:33.116636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.116599 5733 scope.go:122] "RemoveContainer" containerID="1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9" Oct 30 20:04:33.119706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.119684 5733 generic.go:356] "Generic (PLEG): container finished" podID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerID="a1dc7fe9a61003310f76426a30fb6ae8c9d5af188a4d9d6eb9031e015f8badbc" exitCode=0 Oct 30 20:04:33.119801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.119785 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" event={"ID":"1a49222b-1864-4b1f-bf27-15e207bc17f5","Type":"ContainerDied","Data":"a1dc7fe9a61003310f76426a30fb6ae8c9d5af188a4d9d6eb9031e015f8badbc"} Oct 30 20:04:33.129667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.129647 5733 scope.go:122] "RemoveContainer" containerID="1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9" Oct 30 20:04:33.129981 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:04:33.129958 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9\": container with ID starting with 1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9 not found: ID does not exist" containerID="1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9" Oct 30 20:04:33.130102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.129986 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9"} err="failed to get container status \"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9\": rpc error: code = NotFound desc = could not find container \"1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9\": container with ID starting with 1fce3cf0ff67c1e9dd89992816f164686d51c53bc1df0ca70d53b69efbd770c9 not found: ID does not exist" Oct 30 20:04:33.203993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.203963 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204098 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204171 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204210 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204301 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204340 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204403 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204500 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204637 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\") pod \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\" (UID: \"81e3bbd1-0da1-4638-aaec-8bcf01b332a1\") " Oct 30 20:04:33.204711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204652 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.204847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.204828 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.205411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.205379 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.205761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.205738 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.205912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.205889 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.205977 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.205921 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.205977 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.205938 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.206149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.206125 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:33.211264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.211237 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "kube-api-access-t25hc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:33.211697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.211667 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.212173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.211815 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:33.213663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.213635 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo" pod "81e3bbd1-0da1-4638-aaec-8bcf01b332a1" (UID: "81e3bbd1-0da1-4638-aaec-8bcf01b332a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:33.272513 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.272479 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 20:04:33.307065 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307034 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307177 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307221 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307304 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.307420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307405 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307464 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307509 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307512 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.307655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307593 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.307879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307834 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.308013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.307994 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:33.308202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.308154 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.308202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.308169 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\") pod \"1a49222b-1864-4b1f-bf27-15e207bc17f5\" (UID: \"1a49222b-1864-4b1f-bf27-15e207bc17f5\") " Oct 30 20:04:33.308495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.308464 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310709 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310737 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310753 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310767 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310782 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t25hc\" (UniqueName: \"kubernetes.io/projected/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-kube-api-access-t25hc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310796 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310810 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310823 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a49222b-1864-4b1f-bf27-15e207bc17f5-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310837 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81e3bbd1-0da1-4638-aaec-8bcf01b332a1-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310851 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.310865 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.311112 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:33.311592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.311250 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "kube-api-access-4j8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:33.313248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.313227 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:33.336099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.332204 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy" pod "1a49222b-1864-4b1f-bf27-15e207bc17f5" (UID: "1a49222b-1864-4b1f-bf27-15e207bc17f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:33.412101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.412066 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j8zz\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-kube-api-access-4j8zz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.412101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.412095 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.412101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.412107 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.412370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.412116 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a49222b-1864-4b1f-bf27-15e207bc17f5-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:33.638785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:33.638746 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.130747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.130713 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc" Oct 30 20:04:34.133107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.133081 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" Oct 30 20:04:34.133239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.133046 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9" event={"ID":"1a49222b-1864-4b1f-bf27-15e207bc17f5","Type":"ContainerDied","Data":"0ea7ba90d58c332853301177e28380e22f83deabfbf221809aeb6282139a963e"} Oct 30 20:04:34.133296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.133246 5733 scope.go:122] "RemoveContainer" containerID="a1dc7fe9a61003310f76426a30fb6ae8c9d5af188a4d9d6eb9031e015f8badbc" Oct 30 20:04:34.243159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.243115 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 20:04:34.288166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.288115 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-6cbb9"] Oct 30 20:04:34.341140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.333921 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 20:04:34.356084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.351832 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-dftlc"] Oct 30 20:04:34.390705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:34.390667 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.215269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:35.215219 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.040729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:36.040696 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.084078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:36.084030 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" path="/var/lib/kubelet/pods/1a49222b-1864-4b1f-bf27-15e207bc17f5/volumes" Oct 30 20:04:36.084967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:36.084938 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" path="/var/lib/kubelet/pods/81e3bbd1-0da1-4638-aaec-8bcf01b332a1/volumes" Oct 30 20:04:36.997077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:36.997032 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:37.809797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:37.809751 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.786297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:38.786260 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.964600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:38.964563 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.446083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:40.445607 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.257702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:41.257659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.662527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:41.662476 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:41.990753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:41.990638 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.088571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:43.088535 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9"] Oct 30 20:04:43.093025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:43.092992 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4a5e99-fl7s9"] Oct 30 20:04:43.443920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:43.443882 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:44.056673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:44.056636 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c13f63-f778-4d85-9f09-d61e80ad7ad1" path="/var/lib/kubelet/pods/a8c13f63-f778-4d85-9f09-d61e80ad7ad1/volumes" Oct 30 20:04:44.633960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:44.633877 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:45.831936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:45.831897 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.938099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:46.937970 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:47.273046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:47.272949 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.375477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.375437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.534818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.534782 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.800583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.800508 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:04:48.801772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.801752 5733 cpu_manager.go:414] "containerMap: removing container" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" Oct 30 20:04:48.801862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.801779 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" Oct 30 20:04:48.801862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.801850 5733 cpu_manager.go:414] "containerMap: removing container" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" Oct 30 20:04:48.801862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.801858 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" Oct 30 20:04:48.802089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.802070 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e3bbd1-0da1-4638-aaec-8bcf01b332a1" containerName="istio-proxy" Oct 30 20:04:48.802138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.802096 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a49222b-1864-4b1f-bf27-15e207bc17f5" containerName="istio-proxy" Oct 30 20:04:48.806719 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.806700 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.831318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.831286 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19f2ee-openshift-default-dockercfg-fqltd\"" Oct 30 20:04:48.840006 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.839975 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:04:48.899439 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.899386 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:48.938080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.917461 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.938411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.938383 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.938557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.938542 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.938663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.938645 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnx6q\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.952075 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.938791 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.963078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.952289 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.963542 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.963513 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.963685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.963669 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:48.963817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:48.963803 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.070785 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.070838 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.070882 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.086355 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.086637 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.094136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.088388 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104318 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104399 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104439 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104466 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnx6q\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104526 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.104728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.104559 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.108974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.108900 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.109669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.109640 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.135547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.135511 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.150219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.138214 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.150219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.139582 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnx6q\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.162706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.162517 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.163196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.163148 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\") pod \"e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.422260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.422179 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:49.745741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.745706 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:04:49.746410 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:04:49.746377 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e31201_64ff_4e20_999c_7e3be595e84a.slice/crio-4f0bac2ba5fa50c8f237a50bdfdd15b6d1457f8740cd28335adbf774aaea3cc7 WatchSource:0}: Error finding container 4f0bac2ba5fa50c8f237a50bdfdd15b6d1457f8740cd28335adbf774aaea3cc7: Status 404 returned error can't find the container with id 4f0bac2ba5fa50c8f237a50bdfdd15b6d1457f8740cd28335adbf774aaea3cc7 Oct 30 20:04:49.750143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.750100 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:49.750263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.750189 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:49.750263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.750242 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:49.806515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:49.806478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:50.341456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.341415 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerStarted","Data":"b4f1127e96b1587f58efc3755a137dc8b764092b74c96e0e4bcfd26794c0cd4e"} Oct 30 20:04:50.341618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.341482 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerStarted","Data":"4f0bac2ba5fa50c8f237a50bdfdd15b6d1457f8740cd28335adbf774aaea3cc7"} Oct 30 20:04:50.374043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.373991 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podStartSLOduration=2.373975766 podStartE2EDuration="2.373975766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:50.369801166 +0000 UTC m=+2050.437636735" watchObservedRunningTime="2026-10-30 20:04:50.373975766 +0000 UTC m=+2050.441811334" Oct 30 20:04:50.422596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.422535 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:50.428547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.428516 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:50.946145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:50.946079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:51.366081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:51.359236 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:51.378981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:51.378646 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:04:52.138418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:52.138378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.298740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:52.298708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.977641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:52.977600 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.155228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.155194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.326455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.326367 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.493325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.493274 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.805254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.805217 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 20:04:53.805739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.805704 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" containerID="cri-o://3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489" gracePeriod=30 Oct 30 20:04:53.985576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:53.985540 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.307082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.302708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.318082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.308960 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 20:04:54.399888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.399855 5733 generic.go:356] "Generic (PLEG): container finished" podID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerID="3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489" exitCode=0 Oct 30 20:04:54.400113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.399905 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" Oct 30 20:04:54.400113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.399941 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" event={"ID":"0466fa1a-6c5c-4a43-a71d-42b4d000b07c","Type":"ContainerDied","Data":"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489"} Oct 30 20:04:54.400113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.399994 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq" event={"ID":"0466fa1a-6c5c-4a43-a71d-42b4d000b07c","Type":"ContainerDied","Data":"0fe2404807e4ef108de434a8e4ba2eb6417c1021a43a1f54aa55d6745fa26d22"} Oct 30 20:04:54.400113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.400024 5733 scope.go:122] "RemoveContainer" containerID="3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489" Oct 30 20:04:54.463435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.463016 5733 scope.go:122] "RemoveContainer" containerID="3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489" Oct 30 20:04:54.463598 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:04:54.463517 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489\": container with ID starting with 3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489 not found: ID does not exist" containerID="3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489" Oct 30 20:04:54.463660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.463585 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489"} err="failed to get container status \"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489\": rpc error: code = NotFound desc = could not find container \"3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489\": container with ID starting with 3d784542e7556fc1cdf1f94d9abfde95f23a2def2977355264e36bf192bd1489 not found: ID does not exist" Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466278 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466511 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466545 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466578 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466765 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466823 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.466878 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\") pod \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\" (UID: \"0466fa1a-6c5c-4a43-a71d-42b4d000b07c\") " Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.467195 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:54.469472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.467930 5733 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-ca-configmap\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.473199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.473162 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "kube-api-access-vb58c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:54.477935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.477884 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:54.478330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.478298 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:54.478583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.478567 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:54.483216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.483187 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:04:54.483306 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.483198 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs" pod "0466fa1a-6c5c-4a43-a71d-42b4d000b07c" (UID: "0466fa1a-6c5c-4a43-a71d-42b4d000b07c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:54.491650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.491618 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:04:54.492983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.492956 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" Oct 30 20:04:54.493088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.492985 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" Oct 30 20:04:54.493215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.493198 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" containerName="discovery" Oct 30 20:04:54.509046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.505555 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.531852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.521329 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-55d2e0-openshift-default-dockercfg-4ghqw\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.567110 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.579962 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580046 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580099 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580141 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580330 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580379 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580429 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580672 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580711 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dr2v\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580879 5733 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-local-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580906 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb58c\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-kube-api-access-vb58c\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580923 5733 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-cacerts\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580938 5733 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-kubeconfig\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580953 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.601100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.580969 5733 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0466fa1a-6c5c-4a43-a71d-42b4d000b07c-istio-csr-dns-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:54.666726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.652544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697259 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697313 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dr2v\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697379 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697436 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697468 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697507 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697688 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697731 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.699154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.697781 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.761106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.756553 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.825696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.773303 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.826931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.802668 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.835094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.807778 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.835315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.824863 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.835502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.827732 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.872250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.856576 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dr2v\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.914627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.900322 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:54.914627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:54.905572 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\") pod \"e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:55.054336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.054228 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 20:04:55.061843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.061808 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-g9dvq"] Oct 30 20:04:55.062025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.061952 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:55.153096 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.146489 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.460144 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.460112 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:04:55.461454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.461419 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:55.461549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.461500 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:55.461598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.461549 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:04:55.496440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.496406 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.591349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.589878 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:04:55.651448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.650135 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:04:55.651448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.651281 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.659690 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.659661 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19f2ee\"" Oct 30 20:04:55.669690 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.669235 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:04:55.677171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.669668 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.695659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.695581 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19f2ee-dockercfg-jz827\"" Oct 30 20:04:55.701228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.698793 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.762142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.762113 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:04:55.762688 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.762659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.795092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.786201 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thds\" (UniqueName: \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.795092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.786356 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.795092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.786469 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8xwg\" (UniqueName: \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.899172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.888939 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.899172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.889203 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thds\" (UniqueName: \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.902601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.900415 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.904910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.903598 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.904910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.903736 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8xwg\" (UniqueName: \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:55.904910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.904867 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.949078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.939754 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thds\" (UniqueName: \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\") pod \"payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:55.949078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:55.941497 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8xwg\" (UniqueName: \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\") pod \"payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:56.070336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.064345 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:56.099105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.094822 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:56.231084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.210090 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0466fa1a-6c5c-4a43-a71d-42b4d000b07c" path="/var/lib/kubelet/pods/0466fa1a-6c5c-4a43-a71d-42b4d000b07c/volumes" Oct 30 20:04:56.468210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.468166 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerStarted","Data":"0d226d4f4313ad8b338e662dc99348dd946b857c1cedcac66ea275853e13df7d"} Oct 30 20:04:56.468756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.468242 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerStarted","Data":"772f4f37fc184cc69c024aa73f3096742e9f6970168cb03f5063b4651912532c"} Oct 30 20:04:56.524080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.523015 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podStartSLOduration=2.5229954919999997 podStartE2EDuration="2.522995492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:56.513588296 +0000 UTC m=+2056.581423866" watchObservedRunningTime="2026-10-30 20:04:56.522995492 +0000 UTC m=+2056.590831061" Oct 30 20:04:56.582090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.577746 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:04:56.656441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.656412 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:04:56.704821 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:56.704788 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:57.062628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.062537 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:57.069030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.069001 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:57.481189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.481141 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" event={"ID":"8b964b0d-e172-433f-b5cb-7fd19823d2d7","Type":"ContainerStarted","Data":"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c"} Oct 30 20:04:57.481727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.481209 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" event={"ID":"8b964b0d-e172-433f-b5cb-7fd19823d2d7","Type":"ContainerStarted","Data":"7922264d1737215f871ba23437aeed8f2343405453f4e4f12ac9a199d7104f75"} Oct 30 20:04:57.482595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.482569 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:04:57.484681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.484655 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" event={"ID":"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe","Type":"ContainerStarted","Data":"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32"} Oct 30 20:04:57.484792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.484703 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" event={"ID":"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe","Type":"ContainerStarted","Data":"4d67256facb16c6108269fb664dc23fcf0aea64f7320224c448232e3f310c33f"} Oct 30 20:04:57.484792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.484769 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:57.484906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.484798 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:04:57.486016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.485990 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:04:57.506418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.506354 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" podStartSLOduration=2.506337257 podStartE2EDuration="2.506337257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:57.503731164 +0000 UTC m=+2057.571566733" watchObservedRunningTime="2026-10-30 20:04:57.506337257 +0000 UTC m=+2057.574172825" Oct 30 20:04:57.571425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.571362 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podStartSLOduration=2.571343369 podStartE2EDuration="2.571343369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:57.56133526 +0000 UTC m=+2057.629170830" watchObservedRunningTime="2026-10-30 20:04:57.571343369 +0000 UTC m=+2057.639178939" Oct 30 20:04:57.768848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:57.768740 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.165874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:58.165839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.190419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:59.190385 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.381201 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:04:59.381161 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:04:59.450655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:59.450557 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.664425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:04:59.664393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.686782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.686746 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.766743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.766698 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:05:00.772138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.772115 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:00.781177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.781040 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55d2e0\"" Oct 30 20:05:00.790924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.790900 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:05:00.820618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.820579 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:00.820902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.820877 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chjjx\" (UniqueName: \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:00.875453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.875420 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:05:00.880939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.880908 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:00.889185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.889151 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:05:00.903105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.897170 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55d2e0-dockercfg-bm6rt\"" Oct 30 20:05:00.923038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.923000 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr967\" (UniqueName: \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:00.923239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.923138 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:00.923498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.923472 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:00.923498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.923544 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chjjx\" (UniqueName: \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:00.926770 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:00.926745 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:01.035249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.034501 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:01.035249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.034756 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr967\" (UniqueName: \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:01.053077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.041420 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:01.096092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.095899 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:01.114092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.110553 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chjjx\" (UniqueName: \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\") pod \"payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:01.129921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.129708 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr967\" (UniqueName: \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\") pod \"payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:01.239081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.224442 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:01.268029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.267991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:01.389916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.389640 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:01.632647 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:05:01.632606 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229609ef_e005_41c3_807a_54fa69b874d9.slice/crio-e4807336b1f3448b427f98057c91236861fdce26279d0ea67e305c5adc8df3f4 WatchSource:0}: Error finding container e4807336b1f3448b427f98057c91236861fdce26279d0ea67e305c5adc8df3f4: Status 404 returned error can't find the container with id e4807336b1f3448b427f98057c91236861fdce26279d0ea67e305c5adc8df3f4 Oct 30 20:05:01.670510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.670471 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:05:01.734611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:01.734576 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:05:01.740219 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:05:01.740182 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6862aa9e_2401_4f55_b63d_25435bf3eac4.slice/crio-7cd4d270288634355f9503d201a1f2cfdc519dc7193da005df07ae91e6a86bd8 WatchSource:0}: Error finding container 7cd4d270288634355f9503d201a1f2cfdc519dc7193da005df07ae91e6a86bd8: Status 404 returned error can't find the container with id 7cd4d270288634355f9503d201a1f2cfdc519dc7193da005df07ae91e6a86bd8 Oct 30 20:05:02.200353 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.198613 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.564933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.564837 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" event={"ID":"229609ef-e005-41c3-807a-54fa69b874d9","Type":"ContainerStarted","Data":"82879374448bffed3401cd51fab5da854aeef69148f557bff724528cc4ffa351"} Oct 30 20:05:02.564933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.564902 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" event={"ID":"229609ef-e005-41c3-807a-54fa69b874d9","Type":"ContainerStarted","Data":"e4807336b1f3448b427f98057c91236861fdce26279d0ea67e305c5adc8df3f4"} Oct 30 20:05:02.564933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.564932 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:02.567374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.567334 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" event={"ID":"6862aa9e-2401-4f55-b63d-25435bf3eac4","Type":"ContainerStarted","Data":"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5"} Oct 30 20:05:02.567517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.567393 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" event={"ID":"6862aa9e-2401-4f55-b63d-25435bf3eac4","Type":"ContainerStarted","Data":"7cd4d270288634355f9503d201a1f2cfdc519dc7193da005df07ae91e6a86bd8"} Oct 30 20:05:02.567517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.567437 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:02.599198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.599118 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" podStartSLOduration=2.599097686 podStartE2EDuration="2.599097686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:02.593076746 +0000 UTC m=+2062.660912316" watchObservedRunningTime="2026-10-30 20:05:02.599097686 +0000 UTC m=+2062.666933255" Oct 30 20:05:02.622105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:02.619778 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" podStartSLOduration=2.619760286 podStartE2EDuration="2.619760286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:02.615427808 +0000 UTC m=+2062.683263379" watchObservedRunningTime="2026-10-30 20:05:02.619760286 +0000 UTC m=+2062.687595856" Oct 30 20:05:03.129089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:03.118895 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:04.035663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:04.030178 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:05.366282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.366248 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:05.599209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.599176 5733 generic.go:356] "Generic (PLEG): container finished" podID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerID="5952e30aa9cfb6a17ce055ac00ba93d8c9904548c52d436e8214f25ec120003c" exitCode=0 Oct 30 20:05:05.599428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.599245 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerDied","Data":"5952e30aa9cfb6a17ce055ac00ba93d8c9904548c52d436e8214f25ec120003c"} Oct 30 20:05:05.599679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.599662 5733 scope.go:122] "RemoveContainer" containerID="5952e30aa9cfb6a17ce055ac00ba93d8c9904548c52d436e8214f25ec120003c" Oct 30 20:05:05.601693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.601665 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:05.601765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.601750 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:05.601820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:05.601803 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:06.644079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:06.641513 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerStarted","Data":"8ebf90152de528046ce879093a911d81cfb6a204ad85a6d76442badc967fe29d"} Oct 30 20:05:06.698349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:06.697364 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:07.402886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:07.402854 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:05:07.404089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:07.404045 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 20:05:07.404219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:07.404116 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 20:05:07.868648 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:07.868596 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.033081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:08.032032 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.402589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:08.402552 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 20:05:08.402770 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:08.402609 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 20:05:08.450111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:08.450046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.509337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:08.509311 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:05:09.059081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.057307 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:05:09.436090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.422844 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.444075 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.436740 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.464080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.455013 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:05:09.464080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.458045 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\"" Oct 30 20:05:09.474442 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.474396 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 20:05:09.474620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.474464 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 20:05:09.491733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.491695 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:05:09.559988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.559954 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:05:09.567099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567041 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.567366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567345 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.567859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567836 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.567935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567868 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls6g\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.567935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567923 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.568049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.567971 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.568626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.568608 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.569172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.569148 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.569238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.569208 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.671536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.671493 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.671721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.671690 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.671824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.671807 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.671868 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672052 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672364 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672437 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672475 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672500 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls6g\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672567 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.672856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.672654 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.673170 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.673145 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.673925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.673889 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.674869 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.674844 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.676239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.676222 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.677772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.677741 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.681493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.681467 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.687792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.687736 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.699988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.699958 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls6g\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\") pod \"e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:09.791295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:09.788748 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:10.173323 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:05:10.173280 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63919396_52a0_4723_9d83_dd3849acdbc1.slice/crio-fbb8669e9aa14cf72ece6b925664a4340f2cccda92ec8c05fab95d372e6f80a6 WatchSource:0}: Error finding container fbb8669e9aa14cf72ece6b925664a4340f2cccda92ec8c05fab95d372e6f80a6: Status 404 returned error can't find the container with id fbb8669e9aa14cf72ece6b925664a4340f2cccda92ec8c05fab95d372e6f80a6 Oct 30 20:05:10.173982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.173956 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:05:10.177919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.177876 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:10.178032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.177970 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:10.178032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.178016 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:10.287210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.287180 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:10.402294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.402261 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 20:05:10.402449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.402313 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 20:05:10.697899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.697867 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerStarted","Data":"174137be6cb2f13be5c30546e48c66ee0b0c6139b4848b7fe4d1076cebb55397"} Oct 30 20:05:10.698101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.697929 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerStarted","Data":"fbb8669e9aa14cf72ece6b925664a4340f2cccda92ec8c05fab95d372e6f80a6"} Oct 30 20:05:10.751093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.749577 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podStartSLOduration=2.749556877 podStartE2EDuration="2.749556877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:10.735004286 +0000 UTC m=+2070.802839855" watchObservedRunningTime="2026-10-30 20:05:10.749556877 +0000 UTC m=+2070.817392445" Oct 30 20:05:10.789279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:10.789245 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:11.059397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:11.059261 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:11.403014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:11.402985 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 30 20:05:11.403511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:11.403031 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 30 20:05:11.795535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:11.795465 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:12.381992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:12.381949 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.408074 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:12.408029 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:05:12.409243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:12.409220 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:05:12.716438 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:12.716350 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:12.717401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:12.717379 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:05:13.360896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.360859 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.582242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.582205 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:05:13.582720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.582332 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:05:13.704533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.704448 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.828003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.827967 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:05:13.834283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.834259 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.840681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.840649 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:05:13.845028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.845005 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-db02a7-openshift-default-dockercfg-tstcx\"" Oct 30 20:05:13.993604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.993514 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.993778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.993654 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.993843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.993780 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.993934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.993905 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdwt\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.994123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.994095 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.994245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.994213 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.994436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.994402 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.994521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.994452 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:13.994521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:13.994508 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.096616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096576 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.096820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096672 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.096820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096787 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096817 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096872 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.096961 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097011 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097107 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097190 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdwt\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097309 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097599 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097870 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097896 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.097970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.097942 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.099343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.099322 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.099343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.099340 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.110833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.110800 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.110946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.110852 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdwt\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\") pod \"e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.165080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.164769 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:14.187391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.187347 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.524268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.524237 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:05:14.527251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.527180 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:14.527340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.527269 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:14.527340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.527312 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:14.747005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.746954 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" event={"ID":"d421fe44-7234-405c-a2fc-e21240a4f05d","Type":"ContainerStarted","Data":"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135"} Oct 30 20:05:14.747534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.747029 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" event={"ID":"d421fe44-7234-405c-a2fc-e21240a4f05d","Type":"ContainerStarted","Data":"42a9fff551b9201bf2ca7fb1effb3afbe9e76099086a8d15663b1fbf3c911e30"} Oct 30 20:05:14.793292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.793186 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podStartSLOduration=1.793166783 podStartE2EDuration="1.793166783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:14.792303634 +0000 UTC m=+2074.860139202" watchObservedRunningTime="2026-10-30 20:05:14.793166783 +0000 UTC m=+2074.861002353" Oct 30 20:05:14.868613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.868584 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.935283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.935235 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:05:14.942477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.941470 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:14.949036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.949001 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6986a4\"" Oct 30 20:05:14.951200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.949831 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6986a4-dockercfg-lww57\"" Oct 30 20:05:14.954231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.954207 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:05:14.991383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.991349 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:05:14.997882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:14.997852 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.001774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.001732 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:05:15.016609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.014852 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.016609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.015256 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgvj\" (UniqueName: \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.047811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.047725 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.191081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.190430 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:15.201083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.193168 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpw8\" (UniqueName: \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.201083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.193385 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgvj\" (UniqueName: \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.201083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.193780 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.201083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.193875 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.201083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.197724 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.245079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.243204 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:15.245079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.243292 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:15.277081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.253778 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgvj\" (UniqueName: \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\") pod \"payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.280723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.279382 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:15.313129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.298468 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.313129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.298627 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpw8\" (UniqueName: \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.313129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.299993 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.322075 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.319534 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpw8\" (UniqueName: \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\") pod \"payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.332689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.326724 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:15.706080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.704103 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.800509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.795568 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:05:15.952269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:15.952235 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:05:15.953317 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:05:15.953283 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460e7d59_f575_457f_9490_e00297324004.slice/crio-be69da360837380e9d7941f12d5cf36f312a7f3bf20874bf9f25f6d8203f3048 WatchSource:0}: Error finding container be69da360837380e9d7941f12d5cf36f312a7f3bf20874bf9f25f6d8203f3048: Status 404 returned error can't find the container with id be69da360837380e9d7941f12d5cf36f312a7f3bf20874bf9f25f6d8203f3048 Oct 30 20:05:16.165699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.165661 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:16.165876 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.165719 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:16.849511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.849466 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" event={"ID":"460e7d59-f575-457f-9490-e00297324004","Type":"ContainerStarted","Data":"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a"} Oct 30 20:05:16.849978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.849541 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" event={"ID":"460e7d59-f575-457f-9490-e00297324004","Type":"ContainerStarted","Data":"be69da360837380e9d7941f12d5cf36f312a7f3bf20874bf9f25f6d8203f3048"} Oct 30 20:05:16.849978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.849573 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:16.851539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.851506 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" event={"ID":"1349d220-00d7-44a2-84c8-5b1fafb43ec2","Type":"ContainerStarted","Data":"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020"} Oct 30 20:05:16.851698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.851570 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" event={"ID":"1349d220-00d7-44a2-84c8-5b1fafb43ec2","Type":"ContainerStarted","Data":"70168e1c4c60cb038d10d733e78ed6fb2b8184da112522f675e19e308549ddb8"} Oct 30 20:05:16.851698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.851606 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:16.877521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.877475 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" podStartSLOduration=2.8774615199999998 podStartE2EDuration="2.87746152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:16.873521457 +0000 UTC m=+2076.941357025" watchObservedRunningTime="2026-10-30 20:05:16.87746152 +0000 UTC m=+2076.945297088" Oct 30 20:05:16.905565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.905513 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podStartSLOduration=2.905499457 podStartE2EDuration="2.905499457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:16.900360828 +0000 UTC m=+2076.968196396" watchObservedRunningTime="2026-10-30 20:05:16.905499457 +0000 UTC m=+2076.973335056" Oct 30 20:05:16.930079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:16.921513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.165492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:17.165405 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:17.165492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:17.165465 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:18.021090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:18.019364 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:18.166866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:18.166378 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:18.166866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:18.166434 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:19.075077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:19.065197 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:19.166141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:19.166103 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:19.166357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:19.166194 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:19.990614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:19.990561 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:20.165689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:20.165646 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:20.166205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:20.165725 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:20.946145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:20.944208 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:21.166357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:21.166306 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body= Oct 30 20:05:21.166817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:21.166395 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" Oct 30 20:05:21.812484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:21.812445 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:22.171881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:22.171849 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:22.172339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:22.172319 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:22.173175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:22.173151 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:05:22.642804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:22.642768 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.513169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:23.503822 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.416536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:24.416498 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.626348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:24.626312 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:25.610947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:25.610598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:26.546780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:26.546722 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.524253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:27.524218 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.868223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:27.868191 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:05:27.868761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:27.868423 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:05:28.206398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:28.206313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.458875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:28.458696 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.379502 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:05:29.379455 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:05:29.452116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:29.452078 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.597428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:29.597395 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.800701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:29.800612 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.394686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.394652 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:05:30.400915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.400893 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.429799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.429766 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.429995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.429965 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430101 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430348 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430488 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430469 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430602 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430647 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430739 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.430816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.430806 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n6lv\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.432788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.432765 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:05:30.435352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.435328 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f244d-openshift-default-dockercfg-n2fdz\"" Oct 30 20:05:30.532024 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.531993 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532051 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532146 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n6lv\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532216 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532329 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532373 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532432 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532549 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532624 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532632 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532713 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532746 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.532848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.532799 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.533449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.533425 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.534831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.534809 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.535093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.535069 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.547320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.547291 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n6lv\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.548851 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.548826 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\") pod \"e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:30.647164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.646119 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.716618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:30.716579 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:31.027703 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:05:31.027667 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7520762_fb61_4512_875b_f376e9a2bbda.slice/crio-a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c WatchSource:0}: Error finding container a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c: Status 404 returned error can't find the container with id a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c Oct 30 20:05:31.031103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.031072 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:05:31.031212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.031175 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:31.031298 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.031270 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:31.031355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.031318 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:05:31.059847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.059814 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" event={"ID":"e7520762-fb61-4512-875b-f376e9a2bbda","Type":"ContainerStarted","Data":"a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c"} Oct 30 20:05:31.643074 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:31.636470 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.073307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.073216 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" event={"ID":"e7520762-fb61-4512-875b-f376e9a2bbda","Type":"ContainerStarted","Data":"9b3a85d39566bc8ea1d012da80d0c2abe4f538741360b532cad3601336a55e87"} Oct 30 20:05:32.106692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.106631 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podStartSLOduration=2.106618277 podStartE2EDuration="2.106618277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:32.100523441 +0000 UTC m=+2092.168359009" watchObservedRunningTime="2026-10-30 20:05:32.106618277 +0000 UTC m=+2092.174453845" Oct 30 20:05:32.553081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.544827 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.727082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.723632 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:32.727082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.723758 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 20:05:32.727082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:32.723818 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 20:05:33.595288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:33.595246 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:33.744084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:33.735101 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 20:05:33.744084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:33.735197 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 20:05:34.297389 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:34.297217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.727081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:34.720473 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 20:05:34.727081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:34.720532 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 20:05:35.099212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:35.099171 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:35.728082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:35.723508 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 20:05:35.728082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:35.723570 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 20:05:35.928749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:35.928690 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.722459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:36.722420 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" start-of-body= Oct 30 20:05:36.723025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:36.722995 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": dial tcp 10.133.0.183:15021: connect: connection refused" Oct 30 20:05:36.756706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:36.756668 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:37.482282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:37.482245 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:37.872367 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:37.872335 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:37.875188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:37.873841 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:37.878464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:37.878442 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:05:38.290421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:38.290329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:39.099968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:39.099917 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:39.302320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:39.302290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.139577 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:40.139505 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.701648 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:40.701622 5733 kubelet.go:1689] "Image garbage collection succeeded" Oct 30 20:05:40.997636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:40.997542 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.085927 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:42.085886 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.915580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:42.915541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.895926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:43.895891 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:44.392938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:44.392904 5733 scope.go:122] "RemoveContainer" containerID="4727b7d6ef7a4c0ee51e6f9c5403ca2450b0bd0756b75544dedfac5fe7350128" Oct 30 20:05:44.405042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:44.405018 5733 scope.go:122] "RemoveContainer" containerID="23eb08a48417c97e4318fa5dcf9ef634cf0d3a82239f2032759fbda38dbcdb1f" Oct 30 20:05:44.687486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:44.687391 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:45.709229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:45.709194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:46.498477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:46.498437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.535077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:47.534739 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:49.412164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:49.402813 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:50.601117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:50.601073 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:51.503947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:51.503901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.245461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:52.245418 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.900031 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:52.897975 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.996998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:53.996946 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:54.364779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:54.361678 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:55.141008 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:55.140971 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:55.993733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:55.993683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.242565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:56.242527 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:57.211024 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:57.210913 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:58.594303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:58.594256 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:59.384784 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:05:59.384742 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:05:59.532074 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:05:59.531273 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:00.660130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:00.660013 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:00.882525 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:00.882487 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:01.722651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:01.722607 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.874923 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:02.861167 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:03.927589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:03.927512 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.715089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:04.711562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:05.654869 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:05.649437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:06.490915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:06.490823 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:06.691708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:06.691673 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:06.845184 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:06.845150 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:07.401236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:07.401201 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:06:07.401453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:07.401294 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:07.501041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:07.500984 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.440587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:08.440544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:09.832864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:09.832819 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:10.830813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:10.830668 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.765116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.745932 5733 generic.go:356] "Generic (PLEG): container finished" podID="46e31201-64ff-4e20-999c-7e3be595e84a" containerID="b4f1127e96b1587f58efc3755a137dc8b764092b74c96e0e4bcfd26794c0cd4e" exitCode=0 Oct 30 20:06:11.765116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.747116 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerDied","Data":"b4f1127e96b1587f58efc3755a137dc8b764092b74c96e0e4bcfd26794c0cd4e"} Oct 30 20:06:11.765116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.747719 5733 scope.go:122] "RemoveContainer" containerID="b4f1127e96b1587f58efc3755a137dc8b764092b74c96e0e4bcfd26794c0cd4e" Oct 30 20:06:11.765116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.764602 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.777494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.766265 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.777494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.766372 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:11.777494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:11.766416 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:12.645208 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:12.638578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.765978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:12.765936 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerStarted","Data":"df7fcccb106f728c47213a96c26aea93c8643781123b82544c3584bd03aaa087"} Oct 30 20:06:13.433082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:13.429543 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:06:13.433082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:13.429683 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:13.433082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:13.429746 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:13.569089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:13.558755 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.430081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:14.425672 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.430081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:14.425743 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:14.457516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:14.457433 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.772020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:14.770007 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.986745 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:14.986714 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:15.439081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:15.438217 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:15.439081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:15.438302 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:16.166905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.166866 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:16.445131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.443154 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.445131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.443226 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:16.503673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.497486 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:16.703313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.703223 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:06:16.774082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.767599 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.774082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.772237 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:06:16.782164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.780040 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82885f-openshift-default-dockercfg-2ht9s\"" Oct 30 20:06:16.859387 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859346 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859490 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859526 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzv5z\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859755 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859791 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859874 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.859993 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.860182 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.860546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.860215 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962261 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962374 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962404 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzv5z\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962541 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962574 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962624 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.962765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.962745 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963107 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963152 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963154 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963194 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963113 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963527 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.963868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.963606 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.965468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.965440 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.966338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.966313 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.976102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.976050 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzv5z\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:16.976508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:16.976448 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\") pod \"e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:17.101883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.101847 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:17.441087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.423461 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.441087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.423520 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:17.583659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.580474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:17.609358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.602667 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:06:17.671083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.670473 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:17.671083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.670551 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:17.671083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.670583 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:06:17.900095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.898745 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" event={"ID":"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9","Type":"ContainerStarted","Data":"73ead69171fb2a4a47423caf7e8e97fb720d3218ff07168624483c08c392cd5b"} Oct 30 20:06:17.900095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.898803 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" event={"ID":"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9","Type":"ContainerStarted","Data":"2ba1b189a90ec183d9e25100dc2f18746ae26bf246d8fcdbc31b28e49f76408a"} Oct 30 20:06:17.955084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:17.952762 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podStartSLOduration=1.9527452570000001 podStartE2EDuration="1.952745257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:17.934027368 +0000 UTC m=+2138.001862939" watchObservedRunningTime="2026-10-30 20:06:17.952745257 +0000 UTC m=+2138.020580824" Oct 30 20:06:18.102962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.102929 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:18.104846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.104824 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.105098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.105015 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:18.413140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.407486 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:18.425084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.424256 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.425084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:18.424325 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Oct 30 20:06:19.102492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.102457 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.102909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.102509 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:19.359136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.358745 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:19.452078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.451392 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:06:19.453419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.453386 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:06:19.454380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:19.454358 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:06:20.102716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:20.102663 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.102716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:20.102715 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:20.538139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:20.537078 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:21.102339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:21.102303 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.102521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:21.102357 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:21.414130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:21.404962 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:22.002277 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:22.002245 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:22.102406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:22.102355 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:22.102582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:22.102434 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:22.704354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:22.702144 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:23.102611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:23.102584 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:23.102777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:23.102633 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:23.635002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:23.634965 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.102421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:24.102387 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body= Oct 30 20:06:24.102948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:24.102439 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" Oct 30 20:06:24.612470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:24.612431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:25.109157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:25.109123 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:25.109615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:25.109593 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:25.110423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:25.110404 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:06:25.546759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:25.546649 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.150365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:26.150326 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.913336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:26.913288 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:27.897815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:27.897781 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:29.237083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:29.236613 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:29.379970 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:06:29.379935 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:06:30.218772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:30.218723 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.242275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:31.242173 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.090230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:32.090196 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:33.003528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:33.003492 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:33.914810 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:33.914762 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:34.975428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:34.970643 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:35.891111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:35.891015 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.122757 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:37.122721 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:38.265461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:38.265431 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.345106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:39.345052 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.449033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:40.448995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:41.578621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:41.578582 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:41.858759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:41.858716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.800475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:42.800402 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:43.778245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:43.778206 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 20:06:43.778508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:43.778482 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" containerID="cri-o://e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756" gracePeriod=30 Oct 30 20:06:44.169479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:44.169438 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:44.457381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:44.457284 5733 scope.go:122] "RemoveContainer" containerID="edc236e4a0253c53971ca13f1748f6b9d68dc537f2711204bd9de805b2ca9270" Oct 30 20:06:44.469531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:44.469508 5733 scope.go:122] "RemoveContainer" containerID="2fc2d4c247bade70f7523486168270c78ff54d771e531e9726fbc6b2d38066a4" Oct 30 20:06:45.173198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:45.173164 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:46.049589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:46.049534 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.361178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:47.361141 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:48.413225 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:48.412991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:49.100239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.100209 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 20:06:49.145845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.145415 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.145845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.145507 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.145845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.145676 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.145845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.145795 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.146230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.146015 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.146230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.146182 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.146346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.146256 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.146346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.146324 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.146458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.146368 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\") pod \"20c8d7fd-e7fd-462e-99ab-ef682af23939\" (UID: \"20c8d7fd-e7fd-462e-99ab-ef682af23939\") " Oct 30 20:06:49.148297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.148210 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:49.148629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.148577 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:49.148796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.148770 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:49.149204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.149138 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:49.152315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.151536 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:49.152838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.152803 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:06:49.153155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.153128 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:06:49.153751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.153726 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:49.153850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.153764 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln" pod "20c8d7fd-e7fd-462e-99ab-ef682af23939" (UID: "20c8d7fd-e7fd-462e-99ab-ef682af23939"). InnerVolumeSpecName "kube-api-access-wpxln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:49.249329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249290 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249325 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249334 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249344 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249354 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249363 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249378 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20c8d7fd-e7fd-462e-99ab-ef682af23939-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249387 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20c8d7fd-e7fd-462e-99ab-ef682af23939-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.249591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.249397 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxln\" (UniqueName: \"kubernetes.io/projected/20c8d7fd-e7fd-462e-99ab-ef682af23939-kube-api-access-wpxln\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:49.327293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.327198 5733 generic.go:356] "Generic (PLEG): container finished" podID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerID="e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756" exitCode=0 Oct 30 20:06:49.327293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.327275 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" Oct 30 20:06:49.327504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.327278 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" event={"ID":"20c8d7fd-e7fd-462e-99ab-ef682af23939","Type":"ContainerDied","Data":"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756"} Oct 30 20:06:49.327504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.327405 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f" event={"ID":"20c8d7fd-e7fd-462e-99ab-ef682af23939","Type":"ContainerDied","Data":"0cab5f1c4b560b6c2219271b55c05baf6d89af710e737f1a9cb946bd4b5825e7"} Oct 30 20:06:49.327504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.327429 5733 scope.go:122] "RemoveContainer" containerID="e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756" Oct 30 20:06:49.347759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.347730 5733 scope.go:122] "RemoveContainer" containerID="e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756" Oct 30 20:06:49.349108 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:06:49.349076 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756\": container with ID starting with e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756 not found: ID does not exist" containerID="e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756" Oct 30 20:06:49.349228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.349108 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756"} err="failed to get container status \"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756\": rpc error: code = NotFound desc = could not find container \"e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756\": container with ID starting with e68d88808f58f50a24583cc6866bab1c5c7efe6840ec048910477dd0e4a7d756 not found: ID does not exist" Oct 30 20:06:49.365040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.365009 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 20:06:49.372112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.372082 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-m7v4f"] Oct 30 20:06:49.452434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:49.451443 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:50.056955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:50.056905 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" path="/var/lib/kubelet/pods/20c8d7fd-e7fd-462e-99ab-ef682af23939/volumes" Oct 30 20:06:50.506451 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:50.506420 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:51.440281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:51.440187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:51.895346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:51.895310 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.073018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:52.072982 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.298892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:52.298801 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.475385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:52.475349 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:53.831223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:53.828332 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:54.969686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:54.969639 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:55.034367 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:55.034322 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:55.322133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:55.321604 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:06:55.322133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:55.322075 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 20:06:55.954077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:55.951153 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:57.334373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:57.334335 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.630558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:58.625674 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:59.386079 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:06:59.386024 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:06:59.575836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:06:59.575797 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:00.763084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:00.759503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:01.786656 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:01.786620 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.412527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:02.412482 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.570111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:02.570038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:02.724760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:02.724658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.545580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:03.545540 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:04.500658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:04.500618 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:05.318132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:05.318017 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:06.301001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:06.300960 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:07.090300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:07.090267 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:07.898748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:07.898692 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:08.791733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:08.791644 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:09.572283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:09.569934 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:10.454070 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:10.454019 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.382047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:11.382012 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:12.170002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.169965 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:12.332914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.332877 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:07:12.333835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.333813 5733 cpu_manager.go:414] "containerMap: removing container" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" Oct 30 20:07:12.333835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.333834 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" Oct 30 20:07:12.334040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.333981 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c8d7fd-e7fd-462e-99ab-ef682af23939" containerName="istio-proxy" Oct 30 20:07:12.338219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.338198 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.344679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.344659 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f5eee-openshift-default-dockercfg-dwlpb\"" Oct 30 20:07:12.350453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.349528 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:07:12.433081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.432980 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433024 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433233 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433431 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433410 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433467 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433582 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433652 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433719 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.433800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.433783 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lmb\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535104 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535161 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535207 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535389 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lmb\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535551 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535553 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535736 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535667 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.535838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.535822 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536048 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536023 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536098 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536299 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536026 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536540 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.536692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.536643 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.537782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.537755 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.538018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.537995 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.549298 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.549272 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lmb\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.549803 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.549747 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\") pod \"e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.658343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.658304 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:12.959582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.959546 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:07:12.964300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.964262 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:12.964425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.964348 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:12.964425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:12.964396 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:07:13.037001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:13.034592 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:13.881255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:13.881219 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" event={"ID":"fa7be5f2-303b-41de-972c-4a36f84f127d","Type":"ContainerStarted","Data":"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724"} Oct 30 20:07:13.881742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:13.881283 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" event={"ID":"fa7be5f2-303b-41de-972c-4a36f84f127d","Type":"ContainerStarted","Data":"f866320ed1746d019209666b6fad0bc0d9110c8b74fd5354227143ecebb5110b"} Oct 30 20:07:14.021554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:14.021521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:14.659245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:14.659202 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:14.660850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:14.660815 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:14.660990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:14.660891 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:14.887080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:14.886922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:15.658858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:15.658817 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:15.659108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:15.658889 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:15.745714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:15.743785 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:16.531079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:16.525722 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:16.659974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:16.659582 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:16.659974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:16.659640 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:17.305100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:17.301221 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:17.658853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:17.658811 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:17.659387 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:17.658882 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:18.235704 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:18.235663 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:18.659029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:18.658987 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:18.659467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:18.659098 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:19.149084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:19.145727 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:19.658686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:19.658645 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Oct 30 20:07:19.658883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:19.658731 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Oct 30 20:07:19.986142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:19.975560 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:20.665355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:20.665324 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:20.665840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:20.665824 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:20.666630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:20.666606 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:07:20.699128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:20.699052 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podStartSLOduration=8.699036311 podStartE2EDuration="8.699036311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:13.925628996 +0000 UTC m=+2193.993464564" watchObservedRunningTime="2026-10-30 20:07:20.699036311 +0000 UTC m=+2200.766871879" Oct 30 20:07:21.117132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:21.113670 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:21.881003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:21.880967 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:22.746376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:22.745301 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:23.626080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:23.625712 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:24.629813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:24.629754 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.356963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:25.356927 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.316106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:26.315994 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:27.118369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:27.118126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:27.918621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:27.918585 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:28.758999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:28.758960 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:29.382557 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:07:29.382512 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:07:29.611073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:29.611026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:30.504101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:30.494194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:31.619081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:31.611304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:32.479418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:32.479383 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:33.648687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:33.648648 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:34.510917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:34.504922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:35.334408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:35.334373 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:35.523928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:35.523887 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:35.691953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:35.691922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:36.421082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:36.420976 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.385122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:37.385087 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:38.306188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:38.306149 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:39.119582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:39.119547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.170119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:40.167538 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:41.151121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:41.151087 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:41.327866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:41.327833 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.427712 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:42.427679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:43.675108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:43.675050 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:44.446080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:44.443984 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:45.383376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:45.379082 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:46.231753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:46.231721 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:47.091842 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:47.091748 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:47.974776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:47.974736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:48.824646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:48.824608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:49.659020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:49.658984 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:50.453214 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:50.453174 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:51.051017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:51.050974 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:51.221868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:51.221830 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:51.448079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:51.445510 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:52.351998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:52.351961 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:52.529564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:52.529526 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:53.464728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.464695 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:53.794079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.793928 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.887220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.887175 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:07:53.888477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.888442 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.898078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.890488 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.935400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.935360 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:53.974202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:53.974155 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.068816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:54.068178 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:07:54.521112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:54.515995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:55.615077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:55.610543 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:56.959595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:56.959562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:57.972594 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:57.972561 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:58.957888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:58.957839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:59.386310 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:07:59.386247 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:07:59.770269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:07:59.770233 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:00.832860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:00.832616 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:01.689076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:01.684865 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:03.803280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:03.803221 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:04.372552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:03.803329 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:05.137132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:05.137037 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:09.181618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:09.083323 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:09.385702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:09.079466 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:08:09.385702 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:08:09.251684 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.209s" Oct 30 20:08:09.385702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:09.267188 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:09.636958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:09.406641 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:10.527809 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.527757 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" start-of-body= Oct 30 20:08:10.610356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.530741 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" Oct 30 20:08:10.610356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.532681 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:10.905161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.643155 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:10.905161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.643253 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:10.905161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:10.860960 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:11.167877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:11.167790 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:11.168263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:11.168232 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:12.436582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:12.436178 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:13.348219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:13.348175 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:13.570234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:13.570194 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:13.745994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:13.745917 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:13.745994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:13.745967 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.051854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.051746 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.051893 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.051912 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.051963 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052032 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052195 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052260 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052366 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052410 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052421 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052442 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052502 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.165211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.052535 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.326976 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.160:50052\" within 1s: context deadline exceeded" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.371311 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.371327 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.556969 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.557019 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.556971 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.558166 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:14.610114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.558228 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:15.180397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.612141 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:15.180397 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:08:14.718815 5733 logging.go:55] [core] [Channel #5363 SubChannel #5365]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.117:50052", ServerName: "10.133.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.117:50052: operation was canceled" Oct 30 20:08:15.180397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.757535 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:08:15.180397 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:08:14.818042 5733 logging.go:55] [core] [Channel #5364 SubChannel #5366]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.160:50052", ServerName: "10.133.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.160:50052: operation was canceled" Oct 30 20:08:15.180397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:14.845535 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.117:50052\" within 1s: context deadline exceeded" Oct 30 20:08:15.534253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:15.534174 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:17.282860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:17.282807 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:08:18.128108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:18.128032 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:08:59.329554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:08:59.329493 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.483014 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.789194 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.797148 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.810742 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.810976 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.811193 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.818850 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.824396 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.824455 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.831208 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.833626 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.837213 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.844801 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.844859 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.845359 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.845453 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.845576 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:00.857396 5733 logging.go:55] [core] [Channel #5394 SubChannel #5397]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.181:50052", ServerName: "10.133.0.181:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.181:50052: operation was canceled" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:00.868603 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.795038 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.795145 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.798208 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.810941 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.811109 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.811439 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.834915 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.841321 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.841327 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.846519 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.846545 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.846545 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.846597 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.859319 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.051842 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.052361 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.053161 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.053242 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.055535 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.085716 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.825418 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.176:50052: i/o timeout" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.838043 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.838106 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:05.989133 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.384562 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.384721 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.385155 5733 trace.go:236] Trace[862855240]: "Calculate volume metrics of crio-socket for pod openshift-insights/insights-runtime-extractor-nrp79" (30-Oct-2026 20:09:05.082) (total time: 1302ms): Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: Trace[862855240]: [1.302342967s] [1.302342967s] END Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.386529 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" podUID="b36aff52-af76-477f-990d-9900e4c43954" containerName="sbdb" probeResult="failure" output="command timed out" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.386889 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.386932 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.388653 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.388926 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.545248261s: [/var/lib/containers/storage/overlay/d0b5d50eab8b19e55404599138968ce0fab1c9326569960e9d92e888cedcd6f2/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.389121 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.54555521s: [/var/lib/containers/storage/overlay/647f6e15ab9d8a2fa21bbb558fd142446c0cfe2e6f5c0dee4fef5acf26f73c6c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.389372 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.545793077s: [/var/lib/containers/storage/overlay/a840a5cd6aa0fb5297fb88157c11f32ddd501bf0a18cae48dab26fc4aa372220/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.389566 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.391184 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.547578164s: [/var/lib/containers/storage/overlay/e1b840464e8cc07f109fdae53d5b21cfcb787e71fe48d25b2309a942df298177/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.397954 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.164:50052: i/o timeout" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398707 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554352459s: [/var/lib/containers/storage/overlay/bee014615b9b408988440f19b6faa189ef2b61bae25d937c12d3069028c3be73/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398746 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554391785s: [/var/lib/containers/storage/overlay/daf54f36d7a77e8abcdf7c879922fa8c021a06edc1874914adfe9fe904f36c3e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398778 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554424364s: [/var/lib/containers/storage/overlay/dff0193ca02c6ee988742278e73fe0168cccce07ccdca1ed8fc2d50eef368660/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398804 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.55444884s: [/var/lib/containers/storage/overlay/1f0b61116ea9c451f6ac30acdee449ea1dd4035a191ab3113079cfb88442e560/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398835 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554478714s: [/var/lib/containers/storage/overlay/5166e1667cbc146eed846b3d18d0cf1d0cdddd94fa95ea4e0e6a5478771bfc69/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398862 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554504301s: [/var/lib/containers/storage/overlay/72c17898254cf2746559de34a1b1c034516763c97674906452d837e84eb1f0ca/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398885 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.55452583s: [/var/lib/containers/storage/overlay/82e59af079bb130410c2819a2c710dfa478a0e2cd96ddc7d7231f3ffae9b7b1d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.398920 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.554558486s: [/var/lib/containers/storage/overlay/5806b2874a05501563f8bd6700d8b89e89795f78a3e5f4c7127701cf190565dd/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.399153 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.55478953s: [/var/lib/containers/storage/overlay/a0763a17fa4d7cb5bc16912761119b375d2e1389e13aed4fcc7d7b83ffd84153/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.400770 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556400321s: [/var/lib/containers/storage/overlay/2b6217838a24d59a8c3b14ee588bb3f01ae3e8f9f21da862418f608769e02361/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.400805 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556404547s: [/var/lib/containers/storage/overlay/6991016e026940e061649e03169c619cacd757c7241fa9db449ca7cffa74a027/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.400837 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556427994s: [/var/lib/containers/storage/overlay/67423489c62aa836320c31931f5857c362db06e51a2311dc240b2e25d97d27bf/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401148 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556727878s: [/var/lib/containers/storage/overlay/a16ceba910c7aad4ddf34bdbe23c575ae8e7607eed46be10631505ab31ec9e00/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401184 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556766226s: [/var/lib/containers/storage/overlay/2beae6ad824d2ea25277f19427753fe664c51d68b1390c55ef1fd60f37e7d54e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401350 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556930636s: [/var/lib/containers/storage/overlay/d62173e5fb1d530076fd55d3d2532736d91db4a56d995a7d3ca077045456b309/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401384 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556963773s: [/var/lib/containers/storage/overlay/7f48ab2ec6535a0ea0e34a6be4c615b800a188241ef811d1bce93336d48767a6/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401411 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.556989829s: [/var/lib/containers/storage/overlay/0a71bdd898e686cfa07b9e892a5af4bff8a7fa69f854c997a60cbd68f8038b13/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401437 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557014713s: [/var/lib/containers/storage/overlay/7866f36e16259595689f8e0902c4086d6de10de69abfc82a37b61715509a3a11/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401462 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557038187s: [/var/lib/containers/storage/overlay/e0acb70d0e3b074125d1ee2f0ec7f7345b64315cdbf79d691524b50c9f4ffcaa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401487 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557056451s: [/var/lib/containers/storage/overlay/969f5662ed3babbb3c6e0d43de82f5a16203b55f32d1a6c89a9a675075ae3b93/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.401722 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557288073s: [/var/lib/containers/storage/overlay/2d92f726ea47f3df3b33065beb4e7069641bb48e44905ceb520b6589de989643/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.402187 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557749794s: [/var/lib/containers/storage/overlay/86be590af754a03e81e0bd4abba5c383d4e5c4c4ac7aa56f4b624d6f05ec0302/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.402215 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557776527s: [/var/lib/containers/storage/overlay/9c3b25edb36f653173eb3aefbe77c389a63b47945598ba5bdc5801f58cee0cd2/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.402239 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.55779937s: [/var/lib/containers/storage/overlay/6a8d31fdc52a72b2ee9dc8cd45763110e7de98194d9e839b889c651be3f6e6f2/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.402265 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.557823211s: [/var/lib/containers/storage/overlay/a0f59ba95bee855d96d71bc8e36ed95ed04fb74a734b55213bf791c71f875dc5/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.403606 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.403821 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.404031 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.405336 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:06.406752 5733 logging.go:55] [core] [Channel #5419 SubChannel #5420]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: operation was canceled" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.447396 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.602943529s: [/var/lib/containers/storage/overlay/55e375912c4814ab08838d9b47b26246a1e12c1ee41e2ea7de9ab012661d0dc9/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.447440 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.602991169s: [/var/lib/containers/storage/overlay/3393e187510565aca56b4f781fb5858e207986465d6f03cebbdf70c564f01ea7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464184 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.619704515s: [/var/lib/containers/storage/overlay/cbd6baaca214c0ca13638b4e937ef93d2f2f8dedd16fe38fd609866b8c951c43/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464220 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.619742038s: [/var/lib/containers/storage/overlay/c148ada0b2b67e7d79eafdb02a42e39fcbbee9964f3834b4cc1b0d8142c6ecaa/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464599 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.619983156s: [/var/lib/containers/storage/overlay/15e7344c9b7cf06ca8e6e8cbc0d0c42797c84e4806cb24b4694158b2635f1c71/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464638 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.620020924s: [/var/lib/containers/storage/overlay/d798aa167bc1524313aaad5ea7c6618a60a9394a02d500c4fe02ac65729ad8e4/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464663 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.620044051s: [/var/lib/containers/storage/overlay/17450e4d6b29714e4f33da393f90291d73e2600165f7c6399f7b8c1819ee875b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.464694 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.620072331s: [/var/lib/containers/storage/overlay/c66c2ac0aa1ee75b8d5d8440dbb83089d0557ba11738bd65ab09188f6a18f76f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.817864 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973096071s: [/var/lib/containers/storage/overlay/98d3a8cbb12d066be0226919b02826d440474bf0319c8c9f881737ae147a49be/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.817927 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.97316483s: [/var/lib/containers/storage/overlay/a83d528efe3b6c5556e41a70ba33acf3caad5be74024a61e9a9d1e5cae574b88/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818297 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973377178s: [/var/lib/containers/storage/overlay/a008d2b8553ebc9fcfd70c6510406fe90e28af77eb8c5949f1aeec4c179034da/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818347 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973408995s: [/var/lib/containers/storage/overlay/316a408e27639fa88899f57a78b511519bc00550da143a14f5dcd192165961ef/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818384 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973430855s: [/var/lib/containers/storage/overlay/9b25bab1cfc79d2875ac3599827ca0c0e07a265f3a56f0f40f5f96f1947a9220/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818420 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973464897s: [/var/lib/containers/storage/overlay/1a4297cca698362b8c64984910584d414b7f351f06f2407758309b0c83da8975/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818458 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973497606s: [/var/lib/containers/storage/overlay/68b2202da16c4c78b6dea17982287438e8cafd7471ee098cf5ffb5b7860c4bf7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818633 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973674198s: [/var/lib/containers/storage/overlay/c055698381d204f7336a2b34b07d47758da69e3a3ab9c033a7d0a7179665c998/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818670 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973706845s: [/var/lib/containers/storage/overlay/9ede33c6c5bcd933005a35b44bfb77b5121e70baa578162a7f048811c4407ecb/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818705 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973742773s: [/var/lib/containers/storage/overlay/bfbbbc8d4e9dd21cab138d7a097a491637b3461d760e14d01976ace9c4c05494/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818737 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973773385s: [/var/lib/containers/storage/overlay/dc5aa501600901a829c11688dd3826eff3948fa2f7b77ed5c9dee79c0ae98f27/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.818908 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.973940006s: [/var/lib/containers/storage/overlay/5eede4e969d79a65075d2cf8473ecfd09abef4057c239172d99d69a48c454b1d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819138 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.97416676s: [/var/lib/containers/storage/overlay/de504e9746b74da649af347ea4a1c45b9e95429d145528751132c52f284806fb/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819185 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974210546s: [/var/lib/containers/storage/overlay/a7c2602d686a0c7005ef6a82a80472d1fb994dadcdf8cc0735e245de292d0337/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819189 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974213476s: [/var/lib/containers/storage/overlay/cd86ab8aa1e602699d2b7f0b137f750c5e8032198363616ddf837e12b51a6b9a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819217 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974239131s: [/var/lib/containers/storage/overlay/480e2e29572f8140a81eeb0935d9bb2dfa294536f8ddc5aef7ba8acb0cc6c3e6/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819224 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974244327s: [/var/lib/containers/storage/overlay/b794605ca5eafac57660072fe5d443c3397d5afbc9aa0f24623d6d005ac4c35c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819251 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974270107s: [/var/lib/containers/storage/overlay/39666cd4fc5ccbdec5422a65b9fff19e7cfc5fb0f6b42dc64e9a68d770d931da/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819269 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.974285864s: [/var/lib/containers/storage/overlay/c379b22898d7f32f317c8f9b0a3743e10d9e5502dc58f09cffb9e98150e90e9d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819295 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.471260306s: [/var/lib/containers/storage/overlay/c651e6200300fc1fb3affd693c64d443df50afdc453b514e1c471bc8c7021f98/diff /var/log/pods/openshift-ingress_e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2_4aae276e-c30b-44e3-af8d-d88aa479cfb4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819319 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.470220627s: [/var/lib/containers/storage/overlay/0f72861fc0b556d4d3c70373be8f55e20d0d421faa24cf77e4c8cf918e8f5980/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-v484k_99d64cc3-7d77-4972-8df4-4f7040a5de6f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819340 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.854518282s: [/var/lib/containers/storage/overlay/a465a08fa16c83236d866ed0885cbe7f4e36417588e10b43a13a9f94142ec59c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx_8b964b0d-e172-433f-b5cb-7fd19823d2d7/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.819993 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.968751901s: [/var/lib/containers/storage/overlay/4502bc0d86ca3be9bf39c1a00ab228c4c3b53db9a881774ef9340e8efd729535/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820042 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.968804861s: [/var/lib/containers/storage/overlay/08975ddbb9e93360aa3a47b74ac2fe19e59a0fbedce5f3a3ad423638d20682d0/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820093 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.968853977s: [/var/lib/containers/storage/overlay/8c99650f58a49da549e469e868bc6de0db42b3786eddbc7a6d019857f4777eca/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820216 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.968974158s: [/var/lib/containers/storage/overlay/9c665b34584482052b43e75f786e6a69a2c7b019e24ec9a51798bd1cec37494a/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820328 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.969081227s: [/var/lib/containers/storage/overlay/ac8dfb41a960944c713a3701d72536064e95a34f7fadc0d743764378fa744fe3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820396 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.969133284s: [/var/lib/containers/storage/overlay/ab5a13096355f3b5c466c7ea0cb6c0a0bd38e68b21eb387e33d048ba188454de/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820437 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.969176653s: [/var/lib/containers/storage/overlay/049c616bfab68e445e103567649577c19bc62870d3e1cee3e16b4095224f248e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820471 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.96915552s: [/var/lib/containers/storage/overlay/67a1ce925355e2e891c789fd67eab98366bce3fc205e3031f59c69d934b36978/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820476 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.969147446s: [/var/lib/containers/storage/overlay/67f773d397465f312bccf9663f44c37a2a7a02f047c6c12113c053f86d2bb207/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.820633 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.969158032s: [/var/lib/containers/storage/overlay/3e68af768b16985619af888c1046b02b6aafa60c39e2d6d216349d68fe760672/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.828335 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.972979113s: [/var/lib/containers/storage/overlay/fec77ea62120dcdcca6ee6308a32b511c09a4ee01cce76e007e79f60685fe9ff/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.856150 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868285 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.012237563s: [/var/lib/containers/storage/overlay/dfa28435bac9e4b8bd22b8eb87f1a32925ca9c836ac354eaf384e996d504a6f0/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868542 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.012499604s: [/var/lib/containers/storage/overlay/410ee0706286065147fd023256046ef0f4917b578a8f4159f08ca825ecb7e919/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868722 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.012386713s: [/var/lib/containers/storage/overlay/5722d269be5d82394c8fef40234fd1e8162bdd9e981bee21a4b10fe015545d61/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868751 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.012412931s: [/var/lib/containers/storage/overlay/b93361ea7076da060add0090c94d52517e3bbe83f2f6ce7893b836f23e0dd49b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868845 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.012018348s: [/var/lib/containers/storage/overlay/850ec4de062d002e414f2cc72a6a2e7476a036201da5de81ad523dcfd9cc2b5d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.868870 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.010065966s: [/var/lib/containers/storage/overlay/2eef44c1647b1d186929421244fd6e26319c7fe4c53d3432ff386ff5f6475161/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869092 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001649599s: [/var/lib/containers/storage/overlay/3430cea64773c3262487b1476a4760552b75eb1f401b028c4cb866e5af535418/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869119 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001674108s: [/var/lib/containers/storage/overlay/35cc6cecc572043a3fc8494cd9fdae0f47a94d0085d4d9cd20d80acbdc2aa0d3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869278 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001804585s: [/var/lib/containers/storage/overlay/72a021bddc66662ac05b78a2c9af3b7127a45381d3f165536ecbe596c8c4fcc5/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869303 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001827969s: [/var/lib/containers/storage/overlay/e1b50971c3439a04c0a752aff39f656de004f6e928f922e338f9fed13eef4c62/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869328 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001851441s: [/var/lib/containers/storage/overlay/ddd41ea27fe6cf611a6b179df5100da3540e717d27508e35c420111433bc3341/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.869435 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.001941645s: [/var/lib/containers/storage/overlay/0cc34087ccb57b30d6b7e2da3978f769047058ea80440aaa9f5192e0904b6904/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.870684 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.910859 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.067246039s: [/var/lib/containers/storage/overlay/355623f2869f4157a35ea7527cbeb8b6afc97d0d1c6e03de5682e05d82e2ecc3/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.911240 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.067623871s: [/var/lib/containers/storage/overlay/2abb0fb134ee523e2301b530351d2fdbcc7efd4b4b51dcc53736ae97bc23e16f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.912428 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.913149 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.914129 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.914162 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.915344 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.915391 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.915612 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.916145 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.182:50052\" within 1s: context deadline exceeded" Oct 30 20:09:16.575562 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:06.917000 5733 logging.go:55] [core] [Channel #5401 SubChannel #5449]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.182:50052", ServerName: "10.133.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.182:50052: operation was canceled" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.919004 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.919030 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.919357 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.164:50052\" within 1s: context deadline exceeded" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.919454 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.919477 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:06.920660 5733 logging.go:55] [core] [Channel #5395 SubChannel #5452]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.164:50052", ServerName: "10.133.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.164:50052: operation was canceled" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.920771 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.920798 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966214 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098473981s: [/var/lib/containers/storage/overlay/a2a98d1baf891b2f8719c22455abc3d3133f8f618a3554fb445dc848befeed7d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966378 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098490585s: [/var/lib/containers/storage/overlay/e9938068f12be2c0e0a206ea45a1c78cb4d7e51aa4088250fbcbff8cb73c269b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966404 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098512485s: [/var/lib/containers/storage/overlay/e48097ac7b2a8f4e5925d863edea078f2230a1067bdcb874f6e052c7fe2e867e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966428 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098534515s: [/var/lib/containers/storage/overlay/f827e5f8aa0140b85715e474bd1c49ce46e6813b8c1a5ffb023070398e252e9c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966453 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098557138s: [/var/lib/containers/storage/overlay/e5a3fc409b04b16c75b74425dcc7a94baf8be961968339653cf22f641686369c/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966475 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098578401s: [/var/lib/containers/storage/overlay/876883b25230b54af5deddda1a54561a51c83eb6d688100bda8adef546eb1976/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966683 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098772699s: [/var/lib/containers/storage/overlay/2570dc39bf7d21c81b3419a067f220b649b911b1df0ceec4c796805f1f82aa02/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966710 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.09879685s: [/var/lib/containers/storage/overlay/901fc636d38258870d38ebbcfcb1bda2028355fa038c41cc3d18480b7acd2c52/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966736 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.09879121s: [/var/lib/containers/storage/overlay/619fc7c6923c745703ee3c18b71256b3cff0bbb9165aa87da25c83207ed4a89f/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.966759 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.098812545s: [/var/lib/containers/storage/overlay/faf484c3eb74f443dbe2006eef91fd7b91856518989851dcf1a656bce84991c2/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.967582 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.967625 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.967679 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.967699 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.969000 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.100804846s: [/var/lib/containers/storage/overlay/b1d01a2babd671c0f4d1b41f2c9ee3ce9548d0f649481f41154c8109ddb3db88/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.969043 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.100847112s: [/var/lib/containers/storage/overlay/7a92f7f72e5e592a7e81cb7067353be5f8a5cc30f9780b48da1002c2b5d82378/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:06.970102 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.57:50052\" within 1s: context deadline exceeded" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:06.970250 5733 logging.go:55] [core] [Channel #5396 SubChannel #5455]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.57:50052", ServerName: "10.133.0.57:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.57:50052: operation was canceled" Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.031759 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.208306827s: [/var/lib/containers/storage/overlay/ae2b70151fb30b6bfca874819236dd1c2cc84fff5dde2c91218f575383d74dfb/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zj7hb_35a6c5c6-c04a-40b6-a97a-828c0bd2d86c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.031792 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.211123657s: [/var/lib/containers/storage/overlay/1bb4f092e5ad015e2a2111ea77279894bd40e9f4d4d581e6690f60b5a627c37d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v_66363c5c-5f40-47f5-bb62-4b00184dba74/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.031810 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.197069951s: [/var/lib/containers/storage/overlay/df4f6394762baff1144b9d7a7b628d355717bee896eec46c71f01fbe7bdcb37f/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr_30e3bfe8-a903-4119-b1a6-46417f98910f/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367568 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.313307562s: [/var/lib/containers/storage/overlay/f5df0ea15b11ada9e2293550538a108006450cba745a0de2479e7a5813e02089/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367615 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.312875372s: [/var/lib/containers/storage/overlay/510ac7f8d91762e337163d78093c07bc606136e011cd79d0071a2d0e0258327d/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367636 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 31.171143788s: [/var/lib/containers/storage/overlay/6fe1c31581631e93f6acf510a9a55115edf19ae515087245873e6555d2727b5f/diff /var/log/pods/openshift-ingress_e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd_154a804d-f91a-4e31-bd7c-1bdc1e7f29e9/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367651 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.523999811s: [/var/lib/containers/storage/overlay/6fb31e9813f119ca30f25fb22b211b2f302c1a41f4cd3b79d894f435e386824b/diff /var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-dwm4z_7a8bac1e-c263-46f7-97e7-860f85904afc/jobset-operator/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367666 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524006504s: [/var/lib/containers/storage/overlay/52e99bca613012bc41a5e4bb074e09d798c559880b435f8b0d6e23c9c2f3f2dc/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-gdnn8_7489a135-99a1-41d0-8b09-1288c18ecf8b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.367689 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524019743s: [/var/lib/containers/storage/overlay/e91d4725d482d270bf943d948f0980f1cc0afc87c66c219c99cef9cfcace1354/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc_3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368655 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525091155s: [/var/lib/containers/storage/overlay/908bebbf4525c9930efdb8661dfee92cda36ce506daa81da5204fb2a9670873d/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368686 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525096799s: [/var/lib/containers/storage/overlay/2122de6ab244bc1a9b955dad4b5d3999c77dbb27141c5d688613e34d3070e80f/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368701 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525108349s: [/var/lib/containers/storage/overlay/7425028b4f73ba015378ccf71c0af98d8bd446a5103581e3459a7b38d6fb0caa/diff /var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368717 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525121528s: [/var/lib/containers/storage/overlay/ca8e5d2befd18673bc235ccfcf73fe9910413d5cb41914c44c63f6bdeeb0a55e/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368734 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525135254s: [/var/lib/containers/storage/overlay/93230bf5ce6af9cf80071a11874a3e77f65cfd971cffe170da69ba1b8beb01e5/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368747 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.421271534s: [/var/lib/containers/storage/overlay/41c9003eb4ce7e3dfe83bc3ff3602685a94a50887a62fe47a8feb550e8b0a37d/diff /var/log/pods/e2e-models-e2e-worker-w1-a1d885_distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr_d550cab4-61b4-43c5-ae10-4eb9597cc7b0/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368761 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524411976s: [/var/lib/containers/storage/overlay/028e444c8b45a0b21f26608e574cf36bcafd21641f18b49769a098bd7c4e1f00/diff /var/log/pods/openshift-ingress-canary_ingress-canary-bct4d_0678842b-b564-4b68-ae5b-aaffd1680479/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368774 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52441064s: [/var/lib/containers/storage/overlay/58046d9378d92d0692fb638f1283465480e12dfda98c4117b990f1404a4dd793/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/nbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368788 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 29.143973217s: [/var/lib/containers/storage/overlay/2435fc1d58b871cbe00e52576c94da4dc0967b551f4065915a98878fbd71557c/diff /var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368802 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524391031s: [/var/lib/containers/storage/overlay/e01fb8fb6bc2017ad854a78689b2d7e29983c5e8b5aa463c5011912a97725ee4/diff /var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368816 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524399077s: [/var/lib/containers/storage/overlay/56d01027ec2148c3bf775f0b040c353da383e8c445226b60b29350821e745241/diff /var/log/pods/kube-system_global-pull-secret-syncer-jllfr_56834a9b-e103-4f55-b54f-14e73b4fa1db/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368834 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524402909s: [/var/lib/containers/storage/overlay/642ebdf4876eed26e2ea4b311c2525dad49483112c20ae9191a4863e404dc1c8/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-139-103.us-east-2.compute.internal_f224810f96f29df31a3e8747164653d1/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368848 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524415513s: [/var/lib/containers/storage/overlay/797aee25ebb624bc0831e870ecb1320ff691dc45227d4b6a2763be947fcab7c9/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/extractor/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.368866 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524429647s: [/var/lib/containers/storage/overlay/e2db25e4caa5ca969ebdf08215bc1903512941931099b226082c40ec09556a27/diff /var/log/pods/openshift-multus_multus-2hcjh_eed84a66-cd93-4e58-adb3-e097e1b17a14/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369159 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525552819s: [/var/lib/containers/storage/overlay/2cfe011ebd9ae6a1368270084c7dc8aec4d527b27da2f2a292312f80f2603c43/diff /var/log/pods/e2e-models-e2e-worker-w5-278ffa_distinct-2-w5-278ffa-kserve-c69ccf475-jppkh_d142cf77-146e-4719-8e79-001e99aead3c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369184 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52552157s: [/var/lib/containers/storage/overlay/1768924c685e4b87d23b289d11779209411086521fc36df7515c3a614b769d35/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj_51bc38b8-b8b2-434e-819b-e7502f4d4441/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369200 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 33.932616506s: [/var/lib/containers/storage/overlay/b537639028490d06b9fb4abbcbe9d1b2f4e07e822ed9f99c547a16f6fcaede86/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rcvnc_6049f1ce-d07c-4f37-9a3c-dfeff0232cf2/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369214 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525441922s: [/var/lib/containers/storage/overlay/9bd1ba2ff6f6c78ba7cbe586e5dd36bfa5d175366453d2076cb76f4936f3dad4/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-82885f-55777b478d-khbl8_382161a6-f53d-4795-b7c4-005dd3e1ca94/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369228 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525408369s: [/var/lib/containers/storage/overlay/38d72313e754a571559ed480e33d982353b0a900a0cad70033e7ff24c66afe6f/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88_058c7110-65ab-4311-9688-b389e30c5f2a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369242 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524634454s: [/var/lib/containers/storage/overlay/24dfa8e7e93dd5849fd0e2f0d88636f205220464800d88b5fa833e2900323308/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369256 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524633381s: [/var/lib/containers/storage/overlay/33ab0a87f689f1542b5e07726c4ec2196ad1236851e11025239b84b2fea10b4c/diff /var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369271 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.524634829s: [/var/lib/containers/storage/overlay/ecd4d1b146cdabf99864fce66465ab78de5befe508499592022f70e42a684d0e/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-l89fd_651c14dc-17ea-4608-b927-ba1af3ea8b41/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369285 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52492042s: [/var/lib/containers/storage/overlay/3a0e2994a5fc77a8fe59aeec232bc6848c10393371a17887d3e833caa76bb930/diff /var/log/pods/openshift-ingress_e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h_bdd45e19-244b-44e1-87c0-d18e32f1de37/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.369980 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.526307134s: [/var/lib/containers/storage/overlay/8907b2a2a94770d20f8d510ea95d231076d18e5d9a3c36347bd1e3ae7053f043/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz_d05d05bb-294b-4d00-a60f-98fd0d84bfdb/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370001 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52623051s: [/var/lib/containers/storage/overlay/3e25f6b5818c9d7664c64585c3c114ec4044f7efc9c8b0943f5a5dbf256b3d7f/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s_c9306a89-0fc2-44cd-9aa2-96f5ec782c02/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370017 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525206049s: [/var/lib/containers/storage/overlay/e1df21269b56c680209b681bf062d852c2e53652c59c03c61c7957148c732b86/diff /var/log/pods/openshift-ingress_e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f_a74bb1f9-6fa8-4a48-92ef-62b50d195958/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370032 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52556518s: [/var/lib/containers/storage/overlay/ff5481ce839cbebd417a9c94f1aaeba01ef1647b23f5d0e80a63dfb108e9ee7e/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-64d7d576f9-fb9qh_03223b13-85fc-4dda-bf3b-dec632a94c83/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370046 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525720833s: [/var/lib/containers/storage/overlay/b63b5cb8797ad5c2e17e0847ca2cfb5840391e56a38b66f455073066355a8ce9/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-ngwls_63e2b306-7302-41be-92fa-0c0dbeb0e6be/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370075 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525118805s: [/var/lib/containers/storage/overlay/0535997866c26a64d78f4aecf2211c32465b6ea7763785688725dbe8ed21ce7d/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370092 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525134758s: [/var/lib/containers/storage/overlay/ce0546fdc06499cd220770cd25c8004e4411b263f866bb7fb0243c179af7f0e5/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/sbdb/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370106 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525144636s: [/var/lib/containers/storage/overlay/d443fdcb8b6dd63985648f5576839be11b38b50ce827557df31af41b59d563a3/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/exporter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370120 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525156097s: [/var/lib/containers/storage/overlay/028e24f7f2a4cb3bea8c03fd6ea0f7819bffd2cd7ebab78992de71ed7a1e6846/diff /var/log/pods/openshift-network-operator_iptables-alerter-2pmgc_29bace5a-fb35-45e9-a2ce-41a61af06524/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370135 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525170409s: [/var/lib/containers/storage/overlay/809e505e260ebe8b625bffd3a3b63b579377ecb8322f490f7e01df9eca93c746/diff /var/log/pods/openshift-monitoring_metrics-server-7bd8695446-nbt67_38afb684-e999-4c10-9281-62f633431cc6/metrics-server/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370148 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.984159755s: [/var/lib/containers/storage/overlay/1b120743020a1387ad4db47ccb5a61d7b2434e892e971cf29c3f2d47a480d74c/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370161 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525719645s: [/var/lib/containers/storage/overlay/baafef32837f7c0b4418e56028b46214876b7b9af1a90ec35e5e662d13fe5506/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6h66z_9a7e5628-1d20-497b-a673-d28a54b125f2/discovery/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370176 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 10.442335098s: [/var/lib/containers/storage/overlay/31024a7533bdef4a43370a4cf9e77247a4b656d471ebbb1dda045c4c8aaf7337/diff /var/log/pods/openshift-ingress_e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq_8b1adab2-ae8a-4b80-9273-97fb0d04e5ac/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370189 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525545469s: [/var/lib/containers/storage/overlay/237b0cd74bb66d72ec74ceb25a86750dfe4012d38e02e43ca31702b70d79a52e/diff /var/log/pods/openshift-ingress_e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw_4b1744f9-9ed1-4e7b-a359-8a1499160126/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370203 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.52598732s: [/var/lib/containers/storage/overlay/721d79f09217f76171f5e68c5a6f55aa038f691a785c7efa10f893c8a2f38172/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq_0c7a23be-1c57-4b4f-a494-0ff740d58fca/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370217 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518977191s: [/var/lib/containers/storage/overlay/e72402eb7c1275e12d2717e37c4836feed966dd897ced9988b246554c4f1a0b1/diff /var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/dns/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370233 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525764777s: [/var/lib/containers/storage/overlay/ff5ab19ad73cca11b947316277eaa03e0cdd50660953830aa31aa84859695795/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx_88acb5d2-4a06-4b9f-a408-0137dad64956/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370249 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.519005258s: [/var/lib/containers/storage/overlay/f49add07ffcd788d6cc26b4d6914193eeea91d10bb273ee9d4c7516efb27ee51/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370265 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.519020215s: [/var/lib/containers/storage/overlay/9528dfa863d8f54c349e8abeb5199826cacca04ca44bff318f2114da1a6788f5/diff /var/log/pods/openshift-multus_network-metrics-daemon-8nnkd_33381540-cec9-4797-a8b9-be05dd1f13c0/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370278 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.519029327s: [/var/lib/containers/storage/overlay/04edfee9b803991c4f7fa145ca5cefba7c4f1a3e5c9a53868ab1299cc371ab65/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370293 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518946641s: [/var/lib/containers/storage/overlay/16f16e38d632c83f931f5f75f399bf18015371cdf86a673a2fd6912d363182b3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370306 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518961846s: [/var/lib/containers/storage/overlay/9145540bdea9a3c96ee30241738486c54106f1aae15487e0220f7d86de5a3697/diff /var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370320 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.526132382s: [/var/lib/containers/storage/overlay/ab80a0d7a25f78c788c581eb45019ffa3f9791cb8e0c4d3e9d7ace42ffc72215/diff /var/log/pods/e2e-models-e2e-worker-w6-5f933e_simulator-w6-5f933e-kserve-86bf55fd5b-pncbq_5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370334 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525636767s: [/var/lib/containers/storage/overlay/e1d263946dd0c4ca5e584378dcb1f8c445544d364b1b0ea9f5e8ae13194c9aa7/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x_47b19494-6b48-41e6-956a-713a34b0dc10/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370347 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525837184s: [/var/lib/containers/storage/overlay/aa21c62792bc33953c2cd21dfed1409b8ec18ea5ad994d996e6ae07451c4e17c/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_distinct-w2-d5e3e5-kserve-c67646f86-96xsp_4e877fbf-b484-4cfc-93a9-b2e813a364b2/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370361 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518542085s: [/var/lib/containers/storage/overlay/74ed5f850d45ab14b404c50727f92e4030e076a9331ce671fe884d37e9f34527/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w_cb994d3f-ac88-4fff-b9ee-400f5556cafb/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370375 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525910289s: [/var/lib/containers/storage/overlay/0252af765c74679864c17d877aa751eefe6cf9ddbad09c835e899bf31dfaf599/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt_bd91d8c7-be08-4975-b46b-70f110ca1bfc/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370388 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.520276348s: [/var/lib/containers/storage/overlay/f5464b00f60f0c82c89ff3bc406ab74753755b99093e5adab31a7827647b06e1/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-glbct_3bb9e4d1-26a1-4698-9322-d963463664b6/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370405 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.514367429s: [/var/lib/containers/storage/overlay/541d816d7209c577f8fd0be27f4ed51adc18ca9f3b2e0b1b712d599fe33f5941/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370418 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.514378937s: [/var/lib/containers/storage/overlay/3805159434d64aff1ea46c7646d7f4497eef42c0ddb63ef15e7786f0b9f14a66/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370431 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.51438899s: [/var/lib/containers/storage/overlay/c23bd4b52006b5789aac48a6cc6900b02f5d87c5d6db911c533c5b4859882e3b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/northd/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370445 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.514101109s: [/var/lib/containers/storage/overlay/a1107cd7fd3cd031a34dee21ad2e363c4f865333fc846656b9d69dd6604d2dc6/diff /var/log/pods/openshift-dns_node-resolver-jcmd8_ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370460 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.50928151s: [/var/lib/containers/storage/overlay/7ea9f2b1e18efc8821d0d2c51fcd33383de7b22dc1102828d6f13ea5a310a5d4/diff /var/log/pods/openshift-ingress_e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4_b535075b-1639-47bb-b428-7c5c0eb5d4b2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370473 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503002891s: [/var/lib/containers/storage/overlay/342d363d463ab87d01f69cca23c0a75b6b0603ac5c33dc80c2bde12419f5e368/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370488 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503009769s: [/var/lib/containers/storage/overlay/c7e154b51cea889b993c9b5d372cf163100c9296f3c1ce940d92987bd35f851a/diff /var/log/pods/openshift-network-diagnostics_network-check-target-mrpl8_ce385369-c46a-4bdd-800a-8ac5ae39a471/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370503 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.325520345s: [/var/lib/containers/storage/overlay/014f1563fa52d38b34cb2ac176fd125a18c8fca4a87f7b740b6361c80cbbd9f5/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370518 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.514997711s: [/var/lib/containers/storage/overlay/693fcbfdd0bda2b572dd905f87386213d64fbe8bb41506afc01c0da23a006b35/diff /var/log/pods/openshift-ingress_e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6_e7520762-fb61-4512-875b-f376e9a2bbda/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370531 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502803373s: [/var/lib/containers/storage/overlay/319054daf94c26aa0b7c451fb41d5fee89f0fb34516561a7de3f8d956d2a296c/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370544 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502814412s: [/var/lib/containers/storage/overlay/2bf5394c4070c85769f116a50902a4f201b5469d39c7eb6d03ed7a0463a874a3/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370557 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.520442931s: [/var/lib/containers/storage/overlay/a3d5d84b16196139f840cbc2b7b7ca0b1db7a17301ea101856aad25039f55028/diff /var/log/pods/openshift-ingress_e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc_ff87fcd9-988d-4f08-aaf3-86d2ff50bab6/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370571 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502740314s: [/var/lib/containers/storage/overlay/16d4d0bd740886dc2781d247a1fc5758b93f80bfbd933e0f4fc2e384776aa6ee/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370586 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502686655s: [/var/lib/containers/storage/overlay/220e1468aed6d6dfdc06972a2f87a1cf70fc1b71f6079b0ca14a6f4375816683/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-2qlwk_e8b76e60-de88-4c65-b570-fb4772dfbb5b/tuned/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370600 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.50265139s: [/var/lib/containers/storage/overlay/7d5cbafe7194b9430fcfd420ef2ada65830a1fb01b4edfc399250d38937b0a2c/diff /var/log/pods/openshift-ingress_e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h_b9558d5f-e925-43dc-90b4-928cd373c624/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370614 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.520259249s: [/var/lib/containers/storage/overlay/1d885477065e41f9f74b33eff686c3c1bda03e20e1b357c1c8cead2e150515e5/diff /var/log/pods/e2e-models-e2e-worker-w4-3f57c5_distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4_64f65e7a-479c-44cd-9f40-5aea55deb3e1/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370628 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.51881264s: [/var/lib/containers/storage/overlay/5e77f67c43a70bdb70a015274a69b587442d9643ee63c6331f2c1c627e5b1d64/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc_7733aeff-8a16-47ab-ae96-b3ee56b1ef38/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370642 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518975332s: [/var/lib/containers/storage/overlay/bf9a97cd93828c5ea7cc3928d0c262510b94e278f3e08d6a39845f157473c463/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr_b700fa9e-7b5a-42f7-9d54-551c147eccc8/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370656 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.527041351s: [/var/lib/containers/storage/overlay/0f743dc2092fcd2e2efb8238a68face705b584d286c440fb589453036ea82b09/diff /var/log/pods/openshift-ingress_e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb_f117faf1-f824-4851-9ce6-e85ebd8fa690/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370671 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502460536s: [/var/lib/containers/storage/overlay/3038d194b3f4dd8c5b2e67ce1ee77388f7f3b38b021420e835f86c657526ff18/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw_f4c14d49-8d28-428b-80a5-51a881aae5d1/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370684 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502958058s: [/var/lib/containers/storage/overlay/25db7643fb80eee942d42689d53de0d585e52cfbf827d0e3fde717003d74ed3e/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-85768f955d-hxzvg_82727e18-39c9-438c-b335-259eb7281109/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370698 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.525795093s: [/var/lib/containers/storage/overlay/7d44de1f35baed0cb48ff040891238122c3f70632540a9431adc71730b329e88/diff /var/log/pods/e2e-models-e2e-worker-w4-3f57c5_premium-w4-3f57c5-kserve-c49958c8f-85rhc_86e76d74-7775-4b7b-9120-885c993ad626/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370712 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.518894805s: [/var/lib/containers/storage/overlay/4cd6bfa788cc712734ec643ce2791749c621439d9de857cb636c5bafafe02233/diff /var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370726 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.958263375s: [/var/lib/containers/storage/overlay/bd5a9efc23d281b533ec11e2f8680cf233516f3bcde0f03afc43648812c50e18/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370739 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.767332576s: [/var/lib/containers/storage/overlay/d32ce8e422358cc390acf5ca696b8fddeed34d92f19e7715552410d135e03227/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370756 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503078916s: [/var/lib/containers/storage/overlay/90e8881dbf612fd21e0940270c75cb53ed04f1a5255bedaf181c45ae57025973/diff /var/log/pods/openshift-ingress_e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg_51892042-d2ae-4a69-9cc4-e347399249ed/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370769 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.60952377s: [/var/lib/containers/storage/overlay/29b4535f1c1b12b757a66a681741d590f25cb786680a561069ed7d3737cbb3b2/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370782 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.592243508s: [/var/lib/containers/storage/overlay/4ac363fa4deec1786013cd643c91c774e6259217479c2c183a220b47a769a243/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370803 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.369720216s: [/var/lib/containers/storage/overlay/3341736d23442d6c635b476020f9cdb98a5e8a8ab81a302b72e266bdc17d87ef/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370816 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.369729562s: [/var/lib/containers/storage/overlay/0b0574bedc52d86a95c4c57bbc5027c65f482876a71a690fbdc83acad5d52a77/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/prometheus/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370831 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.50309944s: [/var/lib/containers/storage/overlay/cf40e9a6bd7307229527715a77b3d0e3c20df991e5a8983e670506ef000b7a94/diff /var/log/pods/openshift-image-registry_node-ca-7gh2g_ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/node-ca/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370844 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502647317s: [/var/lib/containers/storage/overlay/061d7fb9b35a55d54f9f6521de6492034a63c0c26e925fb26e5898372f1f9010/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb_ddb50a54-7c85-49dc-abfd-d9234707efd1/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370859 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.519194736s: [/var/lib/containers/storage/overlay/e225e7fca28dd374498776f82c1158c224640ae62d2045166894ef70a91c24dc/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg_1fe5eb2e-3136-4690-815f-ead77c1a5384/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370884 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.520213752s: [/var/lib/containers/storage/overlay/a49c683f30142faf7b36fde60beacfefb8f403a5ee75449b1b055a371e951796/diff /var/log/pods/openshift-ingress_e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj_f34c469c-95a6-4134-b889-d20cbe59c1a3/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370898 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.519076885s: [/var/lib/containers/storage/overlay/dbe40c0dff2605fd9a14492d700bd0cb1e74c41c31c76276582f03375e726a42/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t_3433ab80-4a84-4013-80d2-7c7555c470d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:09:47.744188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370912 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502717254s: [/var/lib/containers/storage/overlay/f6746051b4cda06bab23c025632f16d0c078477d15979796d2966c0a804dd244/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7_f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370944 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.305533925s: [/var/lib/containers/storage/overlay/e5640c3037d3a35c2c5af767ec9b9fd268a6154260c7ac39adae8fd4a3220c8d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl_6862aa9e-2401-4f55-b63d-25435bf3eac4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370962 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.526191606s: [/var/lib/containers/storage/overlay/d5288b19a987ece6283670a42ff6d62407915c57a164598878595c014746d51f/diff /var/log/pods/e2e-models-e2e-worker-w3-8a8207_distinct-w3-8a8207-kserve-68f4987c5d-zbc97_11fec734-033b-44bb-a7a4-5a3f81ecddeb/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.370976 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503253986s: [/var/lib/containers/storage/overlay/aa052981fee07675cd25a9d170a1d4ead36c63862700cb514dddb2c8228a3096/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk_33c8e7e1-6494-49a8-b61c-bb3098dcccde/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371034 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.181715292s: [/var/lib/containers/storage/overlay/1a17084cbc53035e47a17e1c8cf6fa5483d1b41f48c241e8cbbbd914cc59d8dd/diff /var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371161 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.502948599s: [/var/lib/containers/storage/overlay/9f9bda789fdb612df4515b72d1054cee40659fb1abe5c64f1fdeaa63b955dbcb/diff /var/log/pods/e2e-models-e2e-worker-w6-5f933e_distinct-w6-5f933e-kserve-594b74cb-9fzpf_d1c5d55c-b95f-4c74-9bbb-4d660b088cfa/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371233 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.618135565s: [/var/lib/containers/storage/overlay/fab476cc812599913bd5bc4517f7761c07f02cf432b2c7e676b48c2cec355b78/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl_229609ef-e005-41c3-807a-54fa69b874d9/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371331 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.471163363s: [/var/lib/containers/storage/overlay/385e8a9c5efa91bdb0807303db453221d27970777f413aac4ee43d28ab60cb34/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m_5c087e21-e692-4eb2-8347-baa42ddc21d3/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371422 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503696387s: [/var/lib/containers/storage/overlay/f1b3bc27cd01da413aea250f0076183d604116e5da53cf1cb6d5151010165db1/diff /var/log/pods/e2e-models-e2e-worker-w3-8a8207_distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8_f3c61292-cd14-4230-80ac-2aaf2d55be7c/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371502 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.503293845s: [/var/lib/containers/storage/overlay/d853ce0f7c8afbc838f0028a269b259ae3d026954ff8217512782e702f3926a1/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt_78000019-d14f-4c89-b1d4-06dfcbd4be4e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371560 5733 trace.go:236] Trace[625691743]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" (30-Oct-2026 20:09:06.227) (total time: 1143ms): Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: Trace[625691743]: [1.143805002s] [1.143805002s] END Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.371925 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.479229 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.833147989s: [/var/lib/containers/storage/overlay/cfe279efbca2b6010f73b462ba93c96837d9e64e3cfe65db6350a708e6ee370f/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p_34580b12-bb45-4ca9-be6c-95b6421e8c0e/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.480779 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.092141998s: [/var/lib/containers/storage/overlay/ed856dc6595fda6d93066d70df0d5ccf723e8cf922f17deb2b6c74afb54ebdd9/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2_21b6051c-d265-4cd2-b08b-b64054880109/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.497738 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.443498633s: [/var/lib/containers/storage/overlay/e8b937bb70c28849a3c976c525d3e884bff89fff6ae659ddeae4261f95b689d0/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc_8ed984b5-27dc-47f6-b3a3-0be88a09763e/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.502143 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.504113 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.504151 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.504367 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.505441 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.505472 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.506426 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.506458 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.508823 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.169:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.508943 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.508966 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:07.509600 5733 logging.go:55] [core] [Channel #5400 SubChannel #5466]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.169:50052", ServerName: "10.133.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.169:50052: operation was canceled" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.509742 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.509769 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.511124 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.511154 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.511561 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" podUID="23ce4b26-7e58-442e-85bc-21bc1535c2fd" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.521371 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.521746 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.160:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:07.521867 5733 logging.go:55] [core] [Channel #5402 SubChannel #5473]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.58:50052", ServerName: "10.133.0.58:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.58:50052: operation was canceled" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.961887 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.55573665s: [/var/lib/containers/storage/overlay/6362475accbc9c6701c585fa522b2349e2d737e809a583886ca4bdab330e8368/diff /var/log/pods/openshift-ingress_e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq_90effab3-a893-40cd-b620-673575f1d679/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.965864 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.967713 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.967746 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.967953 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.967975 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968274 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968332 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968409 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968424 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968624 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.968645 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.969135 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.969229 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.969369 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.969394 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.970050 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.918679034s: [/var/lib/containers/storage/overlay/ce0b5893616fb0df5977f6ef75efdf6c1da67df33482416d3728f19265f389a5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm_dc31087c-8882-44ff-9240-42cf43576885/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.970215 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.152537289s: [/var/lib/containers/storage/overlay/fa19b90cab0806d9f426f3bb9d923a291e151f7040aee2258623aa68e7beee3d/diff /var/log/pods/openshift-ingress_e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr_56d233c6-d1fc-4b59-8998-afe61144f065/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.970448 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.919082107s: [/var/lib/containers/storage/overlay/3e036d721aabdc25130ba677f76e63429b4dd1fa95d522d56d482464ca80894d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq_4b80ef9e-12fc-48a1-a912-891358436d57/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:07.970803 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582168474s: [/var/lib/containers/storage/overlay/f5bc013bf9e786123ac615380d98a862fb6322480a06df414b30e017f26dc5d8/diff /var/log/pods/openshift-ingress_e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs_52995638-b72e-460b-8306-f442c213ff19/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.481376 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.613196337s: [/var/lib/containers/storage/overlay/9276feab5386d24697101985b69449a1565e2a5ecec88b52db47a5da33217db9/diff /var/log/pods/openshift-ingress_e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6_9db2cd8e-f002-40ae-9ca1-66041fdec5bd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.481830 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.481890 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.482425 5733 trace.go:236] Trace[1399697921]: "Calculate volume metrics of multus-daemon-config for pod openshift-multus/multus-2hcjh" (30-Oct-2026 20:09:07.369) (total time: 1112ms): Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: Trace[1399697921]: [1.112743428s] [1.112743428s] END Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.549468 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.549559 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.549704 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.569908 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:08.569948 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:11.597023 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:11.597164 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:11.597505 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:11.850295 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.078810 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.078860 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.151356 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.165033 5733 trace.go:236] Trace[700772775]: "Calculate volume metrics of configmap-metrics-client-ca for pod openshift-monitoring/prometheus-k8s-0" (30-Oct-2026 20:09:08.572) (total time: 3592ms): Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: Trace[700772775]: [3.592808586s] [3.592808586s] END Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.165216 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.165263 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:12.166380 5733 trace.go:236] Trace[61355543]: "Calculate volume metrics of ovnkube-config for pod openshift-ovn-kubernetes/ovnkube-node-h6kbw" (30-Oct-2026 20:09:07.967) (total time: 4198ms): Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: Trace[61355543]: [4.198874922s] [4.198874922s] END Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:13.201724 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:13.216137 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:13.216245 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:13.960300 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:14.778460 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:14.778544 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:14.778829 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:14.779125 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.562948 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.609509 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.888742 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.889276 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.889513 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.889709 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.903573 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.903614 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.904700 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:15.904934 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:16.957515 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:16.957838 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:16.957870 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:16.957910 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:18.733373 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:18.733655 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:18.733699 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:18.733918 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:18.734701 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:19.098827 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:19.098881 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:19.339870 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:24.666866 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:24.666999 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:24.667088 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:24.667160 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.757148 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.757495 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.757721 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.757909 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.758257 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:29.297832 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.008612 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.009074 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.018504 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.629032 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:31.179179 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.52:50052\" within 1s: context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:31.179324 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:31.411863 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.198471 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.198548 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.198692 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.198824 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336810 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336879 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-z86lw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336909 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336940 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19f2ee\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336969 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.336996 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337022 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337050 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337118 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ced832-openshift-default-dockercfg-4thzg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337146 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pr8mc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:20.716883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337174 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337203 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55d2e0-dockercfg-bm6rt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337232 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337258 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337292 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"default-dockercfg-6842t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337322 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337350 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337378 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337404 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3f57c5-dockercfg-m7n92\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337432 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337460 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3d0a78\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337489 5733 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337514 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337543 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6986a4\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337570 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337597 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b0d13a-dockercfg-bfmgl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337624 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jqkdh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337651 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337678 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337705 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f5eee-dockercfg-76ns8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337732 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337758 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.337785 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-278ffa-openshift-default-dockercfg-9bwm7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.430838 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.430896 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f246b-dockercfg-8chtl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.430925 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55d2e0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.430954 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f244d-openshift-default-dockercfg-n2fdz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.430981 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431010 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431037 5733 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431098 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431140 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431169 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431196 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a8de3b-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431257 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82885f-openshift-default-dockercfg-2ht9s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431286 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431314 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431340 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431368 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-klkgs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431395 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431421 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431448 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a1d885\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431474 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"default-dockercfg-fzr2x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431501 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431525 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431550 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431576 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-278ffa\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431603 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431629 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431654 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431680 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-274922\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431705 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431733 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431757 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431784 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431811 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431837 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"default-dockercfg-lb6sq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431863 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431888 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431913 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431940 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-278ffa-dockercfg-cvh8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431969 5733 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.431994 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432018 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"distinct-2-w5-278ffa-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432043 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432081 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432106 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qv2b2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432131 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432154 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432179 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432205 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432228 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432254 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432280 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a7bb6c-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432305 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-925e04\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432333 5733 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432360 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"distinct-w2-d5e3e5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432387 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-2-w3-8a8207-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432422 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432448 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432473 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.432498 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4qwx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.457020 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.457401 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.457722 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.492304 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.158:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.531581 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.532466 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.119:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.532698 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.52:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539729 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"default-dockercfg-zwdzq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539798 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539841 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539895 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539939 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.539976 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540012 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540047 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540108 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540144 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540186 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540226 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540269 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540309 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540347 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540387 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqw8g\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540444 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540485 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"premium-w2-d5e3e5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540543 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4f244d-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540588 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-925e04-dockercfg-tstq2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540626 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540668 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540708 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"premium-w4-3f57c5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540748 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"default-dockercfg-2t28t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540788 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a7bb6c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540828 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ced832\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540869 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540910 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540954 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.540993 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-shlrr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541034 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"simulator-w6-5f933e-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541125 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"distinct-2-w1-a1d885-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541166 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-77b4df-openshift-default-dockercfg-47njs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541205 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541244 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541290 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541330 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541369 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.541409 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.567843 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.117:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617257 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617314 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b0d13a\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617339 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617365 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6b35nveaduvtq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617392 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617423 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617463 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617488 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617512 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617538 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617562 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617601 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617628 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617660 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-55d2e0-openshift-default-dockercfg-4ghqw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617686 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617718 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hg2fd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.580766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617749 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"default-dockercfg-dsn78\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617925 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617954 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617979 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618003 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618028 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a7bb6c-dockercfg-5z5xs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618093 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618128 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618152 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618179 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sh8sg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618203 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-db02a7-openshift-default-dockercfg-tstcx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618227 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618259 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"distinct-w6-5f933e-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618297 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618320 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618344 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19f2ee-dockercfg-jz827\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618367 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d5e3e5-dockercfg-tbc7b\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618392 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618421 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ced832-dockercfg-kxkvg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618447 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-8apo69g0vql64\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618474 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f246b\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618498 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7f7d21\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618520 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618544 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jcv8l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618571 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-77b4df-dockercfg-zvrq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618602 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618625 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gr7dw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618648 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"simulator-w2-d5e3e5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618672 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618698 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f5eee\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618739 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f5eee-openshift-default-dockercfg-dwlpb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618765 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618787 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618811 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618835 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618857 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-w3-8a8207-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618882 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618914 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618937 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618959 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5kn8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.618981 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619004 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-77b4df\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619029 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82885f\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619069 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619103 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a1d885-openshift-default-dockercfg-stn6c\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619129 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619151 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619175 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kjn7n\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619197 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d5e3e5\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619221 5733 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619246 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f78dn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619276 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619301 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619324 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619348 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f933e-openshift-default-dockercfg-dspvq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619371 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82885f-dockercfg-jddkg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619410 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19f2ee-openshift-default-dockercfg-fqltd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619448 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3d0a78-dockercfg-wq8s9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619484 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3d0a78-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619507 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.619531 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.650774 5733 ???:1] "http: TLS handshake error from 10.0.139.103:52480: EOF" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.681628 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.689907 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.178:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.706726 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710706 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.166:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710833 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.165:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.711196 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.176:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.778329 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.779906 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.780613 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.175:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.780710 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.780793 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.780920 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.176:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.781246 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.54:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.781314 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.781345 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.56:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.782389 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.169:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:29.139697 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.783833 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:29.338016 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.783892 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.905366 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3f57c5\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.905415 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.55:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.787096 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.171:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.787669 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.783991 5733 trace.go:236] Trace[302635923]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" (30-Oct-2026 20:09:29.300) (total time: 3483ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[302635923]: [3.48321858s] [3.48321858s] END Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:26.758420 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.906205 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.786976 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.58:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.791390 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.160:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.833746 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.166:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.791554 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.791689 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.908371 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.117:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.874158 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.874189 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.664546 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.875859 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.874211 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.890960 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891019 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7f7d21-dockercfg-9v9wb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891034 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891053 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891114 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891000 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-77b4df-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891129 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891166 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"distinct-w4-3f57c5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891148 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891182 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891198 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891214 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891238 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891276 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6986a4-dockercfg-lww57\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891291 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891308 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891322 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-278ffa-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891338 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891353 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d7a733-openshift-default-dockercfg-dwpvw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891370 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891385 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891399 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891414 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.905026 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b0d13a-openshift-default-dockercfg-b2mpb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.909944 5733 logging.go:55] [core] [Channel #5462 SubChannel #5463]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:50052", ServerName: "10.133.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:41930->10.133.0.51:50052: use of closed network connection" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.796396 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.796473 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.796488 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:31.941465 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:31.942172 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.496123 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.606213517s: [/var/lib/containers/storage/overlay/ec2ffdd1d11500fc7c68150df7441e1028313412cbfc625a146d8028c18a7bfe/diff /var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x4v2z_ff194676-b322-42b3-a970-7db28cc2ac3c/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.509019 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.617905 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.627808791s: [/var/lib/containers/storage/overlay/79a61e77f95f8283abf9c70edb8cd8eae9d12e8c61cfcacee1d4740a5cb0d1c2/diff /var/log/pods/openshift-ingress_e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf_d421fe44-7234-405c-a2fc-e21240a4f05d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.650748 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.880952436s: [/var/lib/containers/storage/overlay/2682bd69d8594dceb34999206d3079972f6ad07e099fab1fa34d82eb5a154b21/diff /var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q9sb8_952eb089-1681-4eac-94d4-9f7465666913/main/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.681570 5733 logging.go:55] [core] [Channel #5391 SubChannel #5408]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.55:50052: i/o timeout" Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.695354 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.35518131s: [/var/lib/containers/storage/overlay/b75c006525acf7d20294db6175e10b625264b48a3a0e232ff3346b386adf3ff6/diff /var/log/pods/openshift-ingress_e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8_f5652aad-c74e-4732-8bd7-1d814d64be92/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.707743 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.125278363s: [/var/lib/containers/storage/overlay/35a2f76f42af08ab5ee196b45a1b21616cae90cccab168a1585a8b7534493e6b/diff /var/log/pods/openshift-ingress_e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h_fa7be5f2-303b-41de-972c-4a36f84f127d/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.709119 5733 trace.go:236] Trace[303878916]: "Calculate volume metrics of model-cache for pod e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" (30-Oct-2026 20:09:21.593) (total time: 11115ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[303878916]: [11.115495473s] [11.115495473s] END Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.709622 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710207 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.720479931s: [/var/lib/containers/storage/overlay/0dc0bb446a4f19e94f53a7422bdbd8ae0ebe9f550923ae7247e92a43fd53a49a/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh_460e7d59-f575-457f-9490-e00297324004/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710238 5733 trace.go:236] Trace[1091082889]: "Calculate volume metrics of etc-sysctl-conf for pod openshift-cluster-node-tuning-operator/tuned-2qlwk" (30-Oct-2026 20:09:25.290) (total time: 7419ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[1091082889]: [7.419559186s] [7.419559186s] END Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710577 5733 trace.go:236] Trace[369218009]: "Calculate volume metrics of kube-api-access-t44mj for pod openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" (30-Oct-2026 20:09:00.864) (total time: 31845ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[369218009]: [31.845790243s] [31.845790243s] END Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710863 5733 trace.go:236] Trace[2036092933]: "Calculate volume metrics of kube-api-access-v862h for pod openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" (30-Oct-2026 20:09:00.864) (total time: 31846ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[2036092933]: [31.846099216s] [31.846099216s] END Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.710988 5733 trace.go:236] Trace[1259029033]: "Calculate volume metrics of kube-api-access-v82q6 for pod openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" (30-Oct-2026 20:09:00.864) (total time: 31846ms): Oct 30 20:10:21.931229 ip-10-0-139-103 kubenswrapper[5733]: Trace[1259029033]: [31.846061456s] [31.846061456s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.711350 5733 trace.go:236] Trace[610221081]: "Calculate volume metrics of kube-api-access-nkwsn for pod openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" (30-Oct-2026 20:09:00.864) (total time: 31846ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[610221081]: [31.846415605s] [31.846415605s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.781141 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.552850218s: [/var/lib/containers/storage/overlay/77c119a9da7b0d59b8a20de88ae305897311f25de34a0d69b17c10f2ae1ac082/diff /var/log/pods/openshift-ingress_e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq_affa7e70-81df-440e-a795-9e0b5ee3ae27/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.781232 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.536403936s: [/var/lib/containers/storage/overlay/cb4376a671e5ce28459b08f9027dfe0c89892e0663a67e5da10800c6a299dc00/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm_1349d220-00d7-44a2-84c8-5b1fafb43ec2/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:30.016895 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.787884 5733 trace.go:236] Trace[564227481]: "Calculate volume metrics of kube-api-access-bfgzw for pod openshift-ingress/payload-processing-5f78687d5b-6slx7" (30-Oct-2026 20:09:29.338) (total time: 3448ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[564227481]: [3.448901096s] [3.448901096s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.791324 5733 logging.go:55] [core] [Channel #5392 SubChannel #5406]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: i/o timeout" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.791529 5733 logging.go:55] [core] [Channel #5477 SubChannel #5491]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.160:50052", ServerName: "10.133.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.160:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.825839 5733 logging.go:55] [core] [Channel #5405 SubChannel #5476]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.158:50052", ServerName: "10.133.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.158:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.833196 5733 logging.go:55] [core] [Channel #5494 SubChannel #5498]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.833328 5733 logging.go:55] [core] [Channel #5403 SubChannel #5497]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.119:50052", ServerName: "10.133.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.119:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.853567 5733 logging.go:55] [core] [Channel #5493 SubChannel #5500]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.117:50052", ServerName: "10.133.0.117:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.117:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.853802 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.12776548s: [/var/lib/containers/storage/overlay/da16d09a9c81539072a230f188f771401d17459040268e1e6663cb9d7fb000fa/diff /var/log/pods/openshift-ingress_e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72_0f49df1b-29d9-48cc-b270-00a78df3439e/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.857451 5733 logging.go:55] [core] [Channel #5409 SubChannel #5504]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.178:50052", ServerName: "10.133.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.178:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:32.873613 5733 logging.go:55] [core] [Channel #5404 SubChannel #5474]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.160:50052", ServerName: "10.133.0.160:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.160:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.874239 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.619165 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.619365 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.619927 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.620010 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.620167 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.620250 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.621519 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.858677306s: [/var/lib/containers/storage/overlay/20b7c9e947bb9e4afc78e6d4fbbf27f1a3c4293ee4bcd670e99f159d8c39627b/diff /var/log/pods/openshift-ingress_e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc_46e31201-64ff-4e20-999c-7e3be595e84a/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622311 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6h66z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622654 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.624221 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.659800 5733 trace.go:236] Trace[254825604]: "Calculate volume metrics of tmp for pod openshift-jobset-operator/jobset-operator-674cd59dc5-dwm4z" (30-Oct-2026 20:09:32.516) (total time: 1143ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[254825604]: [1.143480328s] [1.143480328s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.663664 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.663945 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.664089 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:33.665091 5733 logging.go:55] [core] [Channel #5535 SubChannel #5541]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.816877 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.825463 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" podUID="23ce4b26-7e58-442e-85bc-21bc1535c2fd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.667879 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.834170 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.668898 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:09:32.853354 5733 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://af8413d880a654734978d43b2e4628fb-5097b7c7c1477187.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/odh-ai-gateway-infra/events\": http2: client connection lost" event="&Event{ObjectMeta:{maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p.18e36784eff6ac86 odh-ai-gateway-infra 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:odh-ai-gateway-infra,Name:maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p,UID:34580b12-bb45-4ca9-be6c-95b6421e8c0e,APIVersion:v1,ResourceVersion:58722,FieldPath:spec.containers{maas-api},},Reason:Unhealthy,Message:Readiness probe failed: Get \"https://10.133.0.118:8443/health\": context deadline exceeded,Source:EventSource{Component:kubelet,Host:ip-10-0-139-103.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 20:08:17.282854022 +0000 UTC m=+2257.350689595,LastTimestamp:2026-10-30 20:08:17.282854022 +0000 UTC m=+2257.350689595,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-103.us-east-2.compute.internal,}" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.669099 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.873990 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.669553 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622253 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.670174 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622352 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622496 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.670814 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.622791 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.623969 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.624124 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.649269 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.657334 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.659335 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.659485 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.763327 5733 trace.go:236] Trace[1633362322]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" (30-Oct-2026 20:09:32.680) (total time: 1082ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[1633362322]: [1.082969368s] [1.082969368s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.767712 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.181:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.813031 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.813092 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.813756 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:32.891259 5733 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wsp2p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.835216 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.836883 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.837472 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.837956 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.838419 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.838854 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.839303 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.839749 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.840274 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.840745 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.841215 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.841293 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.841760 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.842250 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.952026 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.959008 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.959386 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.968399 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.982369 5733 trace.go:236] Trace[94190441]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (30-Oct-2026 20:09:00.864) (total time: 33117ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[94190441]: [33.117482371s] [33.117482371s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.984479 5733 trace.go:236] Trace[122397516]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" (30-Oct-2026 20:09:32.693) (total time: 1290ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[122397516]: [1.290676588s] [1.290676588s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.985264 5733 trace.go:236] Trace[451758438]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" (30-Oct-2026 20:09:32.709) (total time: 1275ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[451758438]: [1.275586595s] [1.275586595s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:33.992364 5733 trace.go:236] Trace[568194537]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" (30-Oct-2026 20:09:32.710) (total time: 1281ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[568194537]: [1.281987832s] [1.281987832s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.003622 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.003678 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.007920 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.008259 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.008295 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.008563 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.009267 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.087854 5733 trace.go:236] Trace[834203958]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" (30-Oct-2026 20:09:25.781) (total time: 8306ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[834203958]: [8.306275666s] [8.306275666s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.096322 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.116808 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.176:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.116878 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.116895 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.274212 5733 trace.go:236] Trace[261461106]: "Calculate volume metrics of prometheus-k8s-rulefiles-0 for pod openshift-monitoring/prometheus-k8s-0" (30-Oct-2026 20:09:12.165) (total time: 22109ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[261461106]: [22.109120558s] [22.109120558s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.382513 5733 trace.go:236] Trace[461070338]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" (30-Oct-2026 20:09:32.696) (total time: 1686ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[461070338]: [1.686369297s] [1.686369297s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.470864 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.178:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.643383 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.643456 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.170:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.661569 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.664708 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.664736 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.665870 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.672389 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.672453 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.675581 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.836375 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.836444 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.838527 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.838591 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.840733 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.840805 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.842948 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.843008 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.844302 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.844358 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.844982 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-ngwls" podUID="63e2b306-7302-41be-92fa-0c0dbeb0e6be" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.133.0.33:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845316 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845349 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845517 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845578 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845743 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.389578713s: [/var/lib/containers/storage/overlay/15ca07b286479058323af2be854554c9321bfc7886e4b7b0c6efd97d110c0f4d/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs_9bfc13ba-74ce-4e27-af6f-17b6c1a24c18/maas-api/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845759 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845763 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.002139164s: [/var/lib/containers/storage/overlay/838252c0657fe9a005efc02e678e758cd68b58285dbb77251d29d095608ae3b7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845786 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.002147534s: [/var/lib/containers/storage/overlay/3c282a462f48906fc53cbcdc135523159a10003cc4dd6dca8209a1a76791673b/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.845828 5733 logging.go:55] [core] [Channel #5410 SubChannel #5552]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845864 5733 trace.go:236] Trace[165993782]: "Calculate volume metrics of kube-api-access-kc22l for pod openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" (30-Oct-2026 20:09:01.016) (total time: 33828ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[165993782]: [33.828886875s] [33.828886875s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845877 5733 trace.go:236] Trace[670140430]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" (30-Oct-2026 20:09:00.997) (total time: 33848ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[670140430]: [33.848437722s] [33.848437722s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.845912 5733 trace.go:236] Trace[2144324727]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" (30-Oct-2026 20:09:00.860) (total time: 33985ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[2144324727]: [33.985777176s] [33.985777176s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846117 5733 trace.go:236] Trace[853224092]: "Calculate volume metrics of istio-token for pod openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" (30-Oct-2026 20:09:01.138) (total time: 33707ms): Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: Trace[853224092]: [33.70729395s] [33.70729395s] END Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846236 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" start-of-body= Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846268 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846364 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846666 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.841477 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.469193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.846545 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846801 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.846923 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.851931 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:01.856195 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.847253 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.847433 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:02.811844 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.847867 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:09:34.857305 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="9e7003f06b1ea5205c106874d89b156a76a7cb8edcf241554462f565d4d0c3a4" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.859318 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.859333 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.859352 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.859380 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.859921 5733 logging.go:55] [core] [Channel #5538 SubChannel #5542]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.133.0.2:58976->10.133.0.170:50052: use of closed network connection" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861414 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861448 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861526 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861556 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.861766 5733 logging.go:55] [core] [Channel #5507 SubChannel #5555]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.51:50052", ServerName: "10.133.0.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.51:50052: operation was canceled" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861801 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861827 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.861902 5733 logging.go:55] [core] [Channel #5505 SubChannel #5506]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.178:50052", ServerName: "10.133.0.178:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: failed to write client preface: write tcp 10.133.0.2:49688->10.133.0.178:50052: use of closed network connection" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861927 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861940 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861970 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.861983 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862320 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862368 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862596 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862642 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862757 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.862804 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.868945 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869014 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869030 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.869154 5733 logging.go:55] [core] [Channel #5545 SubChannel #5551]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:50052: operation was canceled" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869160 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869240 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869265 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.869629 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.888996 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.889044 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.891475 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-kms8d" podUID="46dad1ef-96d8-483e-909b-c082e47acbbd" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.891531 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-kms8d" podUID="46dad1ef-96d8-483e-909b-c082e47acbbd" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.908846 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.908894 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.909179 5733 logging.go:55] [core] [Channel #5547 SubChannel #5575]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: operation was canceled" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.909759 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.164:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.939124 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.939163 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.939252 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.158:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.939397 5733 logging.go:55] [core] [Channel #5495 SubChannel #5537]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.158:50052", ServerName: "10.133.0.158:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.158:50052: operation was canceled" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.940249 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" podUID="23ce4b26-7e58-442e-85bc-21bc1535c2fd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.47:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.940788 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.943748 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.946338 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.946388 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.946448 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" podUID="b36aff52-af76-477f-990d-9900e4c43954" containerName="sbdb" probeResult="failure" output="command timed out" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.946460 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-h6kbw" podUID="b36aff52-af76-477f-990d-9900e4c43954" containerName="nbdb" probeResult="failure" output="command timed out" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.947440 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.947858 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.947888 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.947933 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.947954 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.948478 5733 logging.go:55] [core] [Channel #5388 SubChannel #5390]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: i/o timeout" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.948539 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.951228 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.960204 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.968576 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.968604 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.971175 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:34.985167 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:34.988169 5733 logging.go:55] [core] [Channel #5502 SubChannel #5503]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.133.0.2:49514->10.133.0.55:50052: use of closed network connection" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.010170 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.010362 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.015683 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.804918 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 34.961286675s: [/var/lib/containers/storage/overlay/497631e6d1e4a7b4c009aabb04601f918712c1d724fff73f2e4792b15a63476c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb_580aade9-3f41-41ff-a977-4a10a7814421/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.992831 5733 trace.go:236] Trace[1887771071]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/alertmanager-main-0" (30-Oct-2026 20:09:34.946) (total time: 1046ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1887771071]: [1.046451649s] [1.046451649s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.993393 5733 trace.go:236] Trace[1391688984]: "Calculate volume metrics of kube-state-metrics-custom-resource-state-configmap for pod openshift-monitoring/kube-state-metrics-5dc8d7d89b-fx95g" (30-Oct-2026 20:09:34.845) (total time: 1147ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1391688984]: [1.147476386s] [1.147476386s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.993969 5733 trace.go:236] Trace[1214559947]: "Calculate volume metrics of prometheus-k8s-rulefiles-1 for pod openshift-monitoring/prometheus-k8s-0" (30-Oct-2026 20:09:34.945) (total time: 1048ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1214559947]: [1.048167973s] [1.048167973s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.994636 5733 trace.go:236] Trace[1135637248]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" (30-Oct-2026 20:09:34.108) (total time: 1886ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1135637248]: [1.886172374s] [1.886172374s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:35.995759 5733 trace.go:236] Trace[1465737371]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" (30-Oct-2026 20:09:00.864) (total time: 35130ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1465737371]: [35.130926617s] [35.130926617s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.034343 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.034390 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.034437 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.034451 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.034749 5733 trace.go:236] Trace[519772620]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" (30-Oct-2026 20:09:34.950) (total time: 1084ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[519772620]: [1.084370617s] [1.084370617s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037526 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037560 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037604 5733 patch_prober.go:29] interesting pod/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037618 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037645 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.037658 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.057144 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.166568 5733 trace.go:236] Trace[1698689351]: "Calculate volume metrics of metrics-server-audit-profiles for pod openshift-monitoring/metrics-server-7bd8695446-nbt67" (30-Oct-2026 20:09:34.951) (total time: 1214ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1698689351]: [1.214934796s] [1.214934796s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.179941 5733 trace.go:236] Trace[918720029]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" (30-Oct-2026 20:09:34.949) (total time: 1230ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[918720029]: [1.230570182s] [1.230570182s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.181100 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.181212 5733 trace.go:236] Trace[212795462]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" (30-Oct-2026 20:09:34.950) (total time: 1230ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[212795462]: [1.230852764s] [1.230852764s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.214380 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:36.214446 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.008209 5733 trace.go:236] Trace[1896877832]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" (30-Oct-2026 20:09:34.953) (total time: 2055ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1896877832]: [2.055124038s] [2.055124038s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.008610 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.009005 5733 trace.go:236] Trace[1477924322]: "Calculate volume metrics of prometheus-trusted-ca-bundle for pod openshift-monitoring/prometheus-k8s-0" (30-Oct-2026 20:09:35.994) (total time: 1014ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1477924322]: [1.014762726s] [1.014762726s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.097072 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.112132 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.142856 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.143626 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.143655 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.145682 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.145712 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.517437 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.517797 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.165:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.694021 5733 trace.go:236] Trace[242559962]: "Calculate volume metrics of configmap-kubelet-serving-ca-bundle for pod openshift-monitoring/metrics-server-7bd8695446-nbt67" (30-Oct-2026 20:09:36.166) (total time: 1527ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[242559962]: [1.527200394s] [1.527200394s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.728375 5733 trace.go:236] Trace[1409750905]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/openshift-state-metrics-7bd7489957-6d6vf" (30-Oct-2026 20:09:34.952) (total time: 2775ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[1409750905]: [2.77598255s] [2.77598255s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.738767 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.738815 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.743720 5733 trace.go:236] Trace[694032435]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" (30-Oct-2026 20:09:34.950) (total time: 2793ms): Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: Trace[694032435]: [2.793249529s] [2.793249529s] END Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.880229 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.181:50052: i/o timeout" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.966904 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:37.983336 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128779 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128848 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128911 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128931 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128969 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.128986 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.129025 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.129041 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.129336 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.129362 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.143053 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.159:50052\" within 1s: context deadline exceeded" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.143850 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:38.143888 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:38.186386 5733 logging.go:55] [core] [Channel #5588 SubChannel #5596]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.159:50052", ServerName: "10.133.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.159:50052: operation was canceled" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.415809 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.415881 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.415887 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-w3-8a8207-kserve-self-signed-certs\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.415957 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"kube-root-ca.crt\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.415979 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416090 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416146 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416183 5733 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:10:22.975747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416217 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416279 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-ced832\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416287 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416339 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416388 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416404 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z724k\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416456 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416478 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q2kw5\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416519 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416553 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-q9kwd\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416608 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416611 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f5eee-dockercfg-76ns8\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416669 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416696 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"default-dockercfg-zwdzq\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416721 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416764 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"premium-w4-3f57c5-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416796 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-278ffa-openshift-default-dockercfg-9bwm7\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416824 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fbg5p\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416879 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416901 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-4f244d-openshift-default-dockercfg-n2fdz\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416938 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.416962 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.417007 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-82885f-openshift-default-dockercfg-2ht9s\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.417038 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.417077 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55d2e0-dockercfg-bm6rt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.418594 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.447781 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f246b\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.447918 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ced832-openshift-default-dockercfg-4thzg\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448032 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-db02a7-openshift-default-dockercfg-tstcx\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448146 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448220 5733 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448302 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448428 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-77b4df-dockercfg-zvrq6\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448503 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448593 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-svfmj\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448650 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448726 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7f7d21\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448811 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-55d2e0-openshift-default-dockercfg-4ghqw\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448865 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.448978 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8apo69g0vql64\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449089 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3d0a78-dockercfg-wq8s9\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449150 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-d5e3e5\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449233 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a1d885-openshift-default-dockercfg-stn6c\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449567 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449689 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"distinct-2-w3-8a8207-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449769 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449828 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1f5eee\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.449941 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"premium-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450035 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"distinct-w6-5f933e-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450100 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450194 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-jcv8l\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450251 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b0d13a\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450305 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a1d885\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450398 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-ced832-dockercfg-kxkvg\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450479 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-82885f-dockercfg-jddkg\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:39.450566 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gr7dw\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.217679 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-klkgs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.218321 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zqbbj\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.218779 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.219133 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.219162 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.219914 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f5eee-openshift-default-dockercfg-dwlpb\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.220034 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3f57c5-openshift-default-dockercfg-rshxj\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.220780 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-19f2ee-dockercfg-jz827\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.221447 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-5f933e-openshift-default-dockercfg-dspvq\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.221892 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-278ffa\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.222433 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.223051 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6b35nveaduvtq\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.223412 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.223929 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z86lw\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.224419 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.230377 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.230574 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.230836 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-a8de3b-serving-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.231043 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.231298 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-925e04-dockercfg-tstq2\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.231440 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3d0a78\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.231685 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.231985 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.232589 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.232922 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.233080 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.233259 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.233615 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"simulator-w6-5f933e-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.233829 5733 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.238629 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-hg2fd\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239290 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239359 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239438 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239470 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239617 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.239461 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.176:50052\" within 1s: context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240255 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240322 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240369 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-278ffa-serving-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240427 5733 trace.go:236] Trace[735308273]: "iptables ChainExists" (30-Oct-2026 20:09:00.844) (total time: 39395ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[735308273]: [39.395795577s] [39.395795577s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240477 5733 trace.go:236] Trace[230482528]: "iptables ChainExists" (30-Oct-2026 20:09:00.844) (total time: 39395ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[230482528]: [39.39551839s] [39.39551839s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.240866 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241023 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7f7d21-dockercfg-9v9wb\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241175 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241268 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241595 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241727 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241825 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f78dn\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.241837 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"default-dockercfg-fzr2x\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.242117 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.242314 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-b0d13a-openshift-default-dockercfg-b2mpb\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.242348 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.242731 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.243094 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.243248 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.243321 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:40.244545 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:41.228638 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:41.230768 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:41.243870 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.517985 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.517985 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518080 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518217 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518326 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518376 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518531 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a7bb6c\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518561 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518660 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518688 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518909 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"distinct-w4-3f57c5-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.518944 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wsp2p\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519107 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519232 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519253 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn76m\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519316 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519411 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:42.519446 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b0d13a-dockercfg-bfmgl\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:43.987606 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:43.987682 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:43.989050 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"distinct-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:43.989244 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:49.566242 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:49.716426 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.55:50052: i/o timeout" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:53.062049 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:53.062389 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:43.333312 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.074702 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\": awaiting required bookmark event for initial events stream, no events received for 19.21778842s" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:50.669635 5733 reflector.go:1159] "Warning: event bookmark expired" err="pkg/kubelet/config/apiserver.go:66: hasn't received required bookmark event marking the end of initial events stream, received last event 10.425758085s ago" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:49.566180 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\": awaiting required bookmark event for initial events stream, no events received for 15.470765322s" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.456031 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.53:50052: i/o timeout" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.456319 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.772306 5733 patch_prober.go:29] interesting pod/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.772399 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.773848 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.773921 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.773592 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.804869 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:50.753449 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-jobset-operator\"/\"webhook-server-cert\": awaiting required bookmark event for initial events stream, no events received for 15.89835656s" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.805728 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"istio-kubeconfig\": awaiting required bookmark event for initial events stream, no events received for 19.89828454s" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:54.861092 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.177565 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.177591 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.178068 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.561202 5733 trace.go:236] Trace[1764509928]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9" (30-Oct-2026 20:09:34.868) (total time: 21692ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1764509928]: ---"Objects streamed" count:1 21692ms (20:09:56.561) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1764509928]: [21.6929294s] [21.6929294s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.561231 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-d5e3e5-openshift-default-dockercfg-pw4z9\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.561471 5733 trace.go:236] Trace[1702103875]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-thanos-prometheus-http-client-file" (30-Oct-2026 20:09:34.868) (total time: 21693ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1702103875]: ---"Objects streamed" count:1 21693ms (20:09:56.561) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1702103875]: [21.693172537s] [21.693172537s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.561489 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.562288 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6h66z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.562327 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.562847 5733 trace.go:236] Trace[2114474368]: "Reflector WatchList" name:object-"openshift-insights"/"insights-runtime-extractor-sa-dockercfg-zsvdk" (30-Oct-2026 20:09:34.868) (total time: 21694ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[2114474368]: ---"Objects streamed" count:1 21694ms (20:09:56.562) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[2114474368]: [21.694226193s] [21.694226193s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.562862 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zsvdk\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794402 5733 trace.go:236] Trace[141071887]: "Reflector WatchList" name:object-"openshift-ingress"/"istiod-tls" (30-Oct-2026 20:09:34.868) (total time: 21925ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[141071887]: ---"Objects streamed" count:0 21925ms (20:09:56.794) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[141071887]: [21.925699663s] [21.925699663s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794425 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794547 5733 trace.go:236] Trace[1289442380]: "Reflector WatchList" name:object-"openshift-monitoring"/"metrics-server-dockercfg-jqkdh" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1289442380]: ---"Objects streamed" count:1 21887ms (20:09:56.794) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1289442380]: [21.887025119s] [21.887025119s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794554 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-jqkdh\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794707 5733 trace.go:236] Trace[374865128]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-b-77b4df-openshift-default-dockercfg-47njs" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[374865128]: ---"Objects streamed" count:1 21887ms (20:09:56.794) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[374865128]: [21.887103955s] [21.887103955s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794717 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-77b4df-openshift-default-dockercfg-47njs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794779 5733 trace.go:236] Trace[596165346]: "Reflector WatchList" name:object-"openshift-monitoring"/"alertmanager-main-tls-assets-0" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[596165346]: ---"Objects streamed" count:1 21887ms (20:09:56.794) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[596165346]: [21.887023942s] [21.887023942s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794784 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794826 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.794853 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795183 5733 trace.go:236] Trace[1934928668]: "Reflector WatchList" name:object-"openshift-monitoring"/"metrics-client-ca" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1934928668]: ---"Objects streamed" count:1 21887ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1934928668]: [21.887325144s] [21.887325144s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795195 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795276 5733 trace.go:236] Trace[1191121851]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-b-6986a4" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1191121851]: ---"Objects streamed" count:1 21887ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1191121851]: [21.887342158s] [21.887342158s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795282 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6986a4\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795369 5733 trace.go:236] Trace[470554261]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-thanos-sidecar-tls" (30-Oct-2026 20:09:34.945) (total time: 21849ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[470554261]: ---"Objects streamed" count:1 21849ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[470554261]: [21.849920183s] [21.849920183s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795375 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795415 5733 trace.go:236] Trace[1444396702]: "Reflector WatchList" name:object-"openshift-ingress"/"istio-kubeconfig" (30-Oct-2026 20:09:34.907) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1444396702]: ---"Objects streamed" count:0 21887ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1444396702]: [21.887970415s] [21.887970415s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795424 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795523 5733 trace.go:236] Trace[361962831]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s" (30-Oct-2026 20:09:34.908) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[361962831]: ---"Objects streamed" count:1 21887ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[361962831]: [21.887405523s] [21.887405523s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795531 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795642 5733 trace.go:236] Trace[86348070]: "Reflector WatchList" name:object-"openshift-monitoring"/"openshift-state-metrics-dockercfg-qv2b2" (30-Oct-2026 20:09:34.947) (total time: 21848ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[86348070]: ---"Objects streamed" count:1 21848ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[86348070]: [21.848057083s] [21.848057083s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795649 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-qv2b2\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795740 5733 trace.go:236] Trace[839954430]: "Reflector WatchList" name:object-"openshift-multus"/"cni-copy-resources" (30-Oct-2026 20:09:34.868) (total time: 21926ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[839954430]: ---"Objects streamed" count:1 21926ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[839954430]: [21.926994945s] [21.926994945s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795747 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795834 5733 trace.go:236] Trace[490380645]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-b-19f2ee-openshift-default-dockercfg-fqltd" (30-Oct-2026 20:09:34.472) (total time: 22323ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[490380645]: ---"Objects streamed" count:1 22323ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[490380645]: [22.323777628s] [22.323777628s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795842 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-19f2ee-openshift-default-dockercfg-fqltd\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795904 5733 trace.go:236] Trace[652840481]: "Reflector WatchList" name:object-"openshift-monitoring"/"alertmanager-main-cluster-tls-config" (30-Oct-2026 20:09:34.868) (total time: 21927ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[652840481]: ---"Objects streamed" count:1 21927ms (20:09:56.795) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[652840481]: [21.927087032s] [21.927087032s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.795911 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796426 5733 trace.go:236] Trace[1768995711]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-worker-w4-3f57c5" (30-Oct-2026 20:09:34.867) (total time: 21623ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1768995711]: ---"Objects streamed" count:1 21623ms (20:09:56.491) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1768995711]: [21.623532195s] [21.623532195s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796442 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3f57c5\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796509 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796768 5733 trace.go:236] Trace[824806588]: "Reflector WatchList" name:object-"cert-manager"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.869) (total time: 21926ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[824806588]: ---"Objects streamed" count:1 21926ms (20:09:56.796) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[824806588]: [21.926913163s] [21.926913163s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796779 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796834 5733 trace.go:236] Trace[243775914]: "Reflector WatchList" name:object-"openshift-multus"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.948) (total time: 21848ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[243775914]: ---"Objects streamed" count:1 21848ms (20:09:56.796) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[243775914]: [21.848699484s] [21.848699484s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796840 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796945 5733 trace.go:236] Trace[572987828]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w1-a1d885"/"distinct-2-w1-a1d885-kserve-self-signed-certs" (30-Oct-2026 20:09:34.870) (total time: 21926ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[572987828]: ---"Objects streamed" count:1 21926ms (20:09:56.796) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[572987828]: [21.926837445s] [21.926837445s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.796952 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"distinct-2-w1-a1d885-kserve-self-signed-certs\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797028 5733 trace.go:236] Trace[1097543957]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w3-8a8207"/"kube-root-ca.crt" (30-Oct-2026 20:09:34.909) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1097543957]: ---"Objects streamed" count:1 21887ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1097543957]: [21.887725683s] [21.887725683s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797034 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797153 5733 trace.go:236] Trace[330746028]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-web-config" (30-Oct-2026 20:09:34.949) (total time: 21847ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[330746028]: ---"Objects streamed" count:1 21847ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[330746028]: [21.847665951s] [21.847665951s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797162 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797233 5733 trace.go:236] Trace[1993540588]: "Reflector WatchList" name:object-"openshift-insights"/"kube-rbac-proxy" (30-Oct-2026 20:09:34.870) (total time: 21926ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1993540588]: ---"Objects streamed" count:1 21926ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1993540588]: [21.926860424s] [21.926860424s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797240 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797339 5733 trace.go:236] Trace[1399549930]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"metrics-server-cert" (30-Oct-2026 20:09:34.909) (total time: 21887ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1399549930]: ---"Objects streamed" count:1 21887ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1399549930]: [21.887894471s] [21.887894471s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797346 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797431 5733 trace.go:236] Trace[1609136979]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-b-d7a733-openshift-default-dockercfg-dwpvw" (30-Oct-2026 20:09:34.949) (total time: 21847ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1609136979]: ---"Objects streamed" count:1 21847ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1609136979]: [21.847456231s] [21.847456231s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797439 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d7a733-openshift-default-dockercfg-dwpvw\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797525 5733 trace.go:236] Trace[1933210730]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-dockercfg-47lpr" (30-Oct-2026 20:09:34.872) (total time: 21925ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1933210730]: ---"Objects streamed" count:1 21925ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1933210730]: [21.925076462s] [21.925076462s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797531 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797599 5733 trace.go:236] Trace[1633654137]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-b-77b4df" (30-Oct-2026 20:09:34.940) (total time: 21857ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1633654137]: ---"Objects streamed" count:1 21857ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1633654137]: [21.857469533s] [21.857469533s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797605 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-77b4df\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797667 5733 trace.go:236] Trace[1449769075]: "Reflector WatchList" name:object-"opendatahub-operator-system"/"kube-root-ca.crt" (30-Oct-2026 20:09:35.021) (total time: 21775ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1449769075]: ---"Objects streamed" count:1 21775ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1449769075]: [21.775912226s] [21.775912226s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797673 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797763 5733 trace.go:236] Trace[1991505396]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-inf-b-6986a4-dockercfg-lww57" (30-Oct-2026 20:09:34.875) (total time: 21922ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1991505396]: ---"Objects streamed" count:1 21922ms (20:09:56.797) Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1991505396]: [21.922041499s] [21.922041499s] END Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797769 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6986a4-dockercfg-lww57\"" Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797855 5733 trace.go:236] Trace[1312861726]: "Reflector WatchList" name:object-"openshift-multus"/"metrics-daemon-secret" (30-Oct-2026 20:09:34.940) (total time: 21857ms): Oct 30 20:10:23.380173 ip-10-0-139-103 kubenswrapper[5733]: Trace[1312861726]: ---"Objects streamed" count:1 21857ms (20:09:56.797) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1312861726]: [21.857598907s] [21.857598907s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797861 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797933 5733 trace.go:236] Trace[521824212]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"kube-root-ca.crt" (30-Oct-2026 20:09:34.891) (total time: 21906ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[521824212]: ---"Objects streamed" count:1 21906ms (20:09:56.797) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[521824212]: [21.906224183s] [21.906224183s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797939 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797988 5733 trace.go:236] Trace[1390369120]: "Reflector WatchList" name:object-"openshift-monitoring"/"metrics-server-client-certs" (30-Oct-2026 20:09:34.940) (total time: 21857ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1390369120]: ---"Objects streamed" count:1 21857ms (20:09:56.797) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1390369120]: [21.857636528s] [21.857636528s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.797994 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798079 5733 trace.go:236] Trace[1186956479]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w4-3f57c5"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.892) (total time: 21906ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1186956479]: ---"Objects streamed" count:1 21906ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1186956479]: [21.90602229s] [21.90602229s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798088 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798197 5733 trace.go:236] Trace[348947594]: "Reflector WatchList" name:object-"openshift-monitoring"/"kube-state-metrics-dockercfg-pr8mc" (30-Oct-2026 20:09:34.891) (total time: 21906ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[348947594]: ---"Objects streamed" count:1 21906ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[348947594]: [21.906341879s] [21.906341879s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798204 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pr8mc\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798296 5733 trace.go:236] Trace[1904376931]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-dockercfg-f5kn8" (30-Oct-2026 20:09:34.940) (total time: 21857ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1904376931]: ---"Objects streamed" count:1 21857ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1904376931]: [21.857748753s] [21.857748753s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798303 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f5kn8\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798405 5733 trace.go:236] Trace[1244365359]: "Reflector WatchList" name:object-"openshift-monitoring"/"node-exporter-dockercfg-ch4qh" (30-Oct-2026 20:09:34.892) (total time: 21906ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1244365359]: ---"Objects streamed" count:1 21906ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1244365359]: [21.90625747s] [21.90625747s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798415 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ch4qh\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798501 5733 trace.go:236] Trace[944013183]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn" (30-Oct-2026 20:09:35.803) (total time: 20995ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[944013183]: ---"Objects streamed" count:1 20995ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[944013183]: [20.995035215s] [20.995035215s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798508 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1f246b-openshift-default-dockercfg-nb8cn\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798607 5733 trace.go:236] Trace[63916165]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w4-3f57c5-dockercfg-m7n92" (30-Oct-2026 20:09:34.892) (total time: 21906ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[63916165]: ---"Objects streamed" count:1 21906ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[63916165]: [21.906374144s] [21.906374144s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798626 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3f57c5-dockercfg-m7n92\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798685 5733 trace.go:236] Trace[1016661827]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-b-55d2e0" (30-Oct-2026 20:09:34.943) (total time: 21855ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1016661827]: ---"Objects streamed" count:1 21855ms (20:09:56.798) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1016661827]: [21.855654015s] [21.855654015s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.798691 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55d2e0\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.805879 5733 trace.go:236] Trace[1698988998]: "Reflector WatchList" name:object-"kube-system"/"konnectivity-agent-dockercfg-p6d49" (30-Oct-2026 20:09:35.803) (total time: 21002ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1698988998]: ---"Objects streamed" count:1 21002ms (20:09:56.805) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1698988998]: [21.002202985s] [21.002202985s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.805897 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p6d49\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.805996 5733 trace.go:236] Trace[720546379]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w5-278ffa"/"default-dockercfg-2t28t" (30-Oct-2026 20:09:34.943) (total time: 21862ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[720546379]: ---"Objects streamed" count:1 21862ms (20:09:56.805) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[720546379]: [21.862781237s] [21.862781237s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806005 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"default-dockercfg-2t28t\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806179 5733 trace.go:236] Trace[1286311787]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w3-8a8207"/"default-dockercfg-lb6sq" (30-Oct-2026 20:09:34.892) (total time: 21913ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1286311787]: ---"Objects streamed" count:1 21913ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1286311787]: [21.913766531s] [21.913766531s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806192 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8a8207\"/\"default-dockercfg-lb6sq\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806374 5733 trace.go:236] Trace[1658103856]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-e2e-inf-b-77b4df-serving-cert" (30-Oct-2026 20:09:36.032) (total time: 20773ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1658103856]: ---"Objects streamed" count:1 20773ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1658103856]: [20.773712964s] [20.773712964s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806386 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-77b4df-serving-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806463 5733 trace.go:236] Trace[688067449]: "Reflector WatchList" name:object-"openshift-cluster-csi-drivers"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.943) (total time: 21862ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[688067449]: ---"Objects streamed" count:1 21862ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[688067449]: [21.862659486s] [21.862659486s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806472 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806553 5733 trace.go:236] Trace[427717092]: "Reflector WatchList" name:object-"kube-system"/"konnectivity-ca-bundle" (30-Oct-2026 20:09:34.892) (total time: 21914ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[427717092]: ---"Objects streamed" count:1 21914ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[427717092]: [21.914067708s] [21.914067708s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806559 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806609 5733 trace.go:236] Trace[909404731]: "Reflector WatchList" name:object-"openshift-multus"/"multus-daemon-config" (30-Oct-2026 20:09:36.032) (total time: 20773ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[909404731]: ---"Objects streamed" count:1 20773ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[909404731]: [20.773821992s] [20.773821992s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806614 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806683 5733 trace.go:236] Trace[249068532]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-metrics-tls" (30-Oct-2026 20:09:34.892) (total time: 21914ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[249068532]: ---"Objects streamed" count:1 21914ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[249068532]: [21.914118393s] [21.914118393s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806689 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806739 5733 trace.go:236] Trace[229675926]: "Reflector WatchList" name:object-"openshift-monitoring"/"metrics-server-audit-profiles" (30-Oct-2026 20:09:34.944) (total time: 21862ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[229675926]: ---"Objects streamed" count:1 21862ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[229675926]: [21.862032316s] [21.862032316s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806744 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806941 5733 trace.go:236] Trace[928520287]: "Reflector WatchList" name:k8s.io/client-go/informers/factory.go:161 (30-Oct-2026 20:09:34.892) (total time: 21914ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[928520287]: ---"Objects streamed" count:1 21914ms (20:09:56.806) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[928520287]: [21.914286372s] [21.914286372s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.806951 5733 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.807262 5733 trace.go:236] Trace[1861189822]: "Reflector WatchList" name:object-"opendatahub-operator-system"/"opendatahub-operator-controller-manager-service-cert" (30-Oct-2026 20:09:36.032) (total time: 20774ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1861189822]: ---"Objects streamed" count:1 20774ms (20:09:56.807) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1861189822]: [20.774386959s] [20.774386959s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.807271 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.807349 5733 trace.go:236] Trace[1982584595]: "Reflector WatchList" name:object-"openshift-network-operator"/"openshift-service-ca.crt" (30-Oct-2026 20:09:36.032) (total time: 20774ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1982584595]: ---"Objects streamed" count:1 20774ms (20:09:56.807) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1982584595]: [20.774341115s] [20.774341115s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.807356 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.879880 5733 trace.go:236] Trace[1326998532]: "Reflector WatchList" name:object-"kuadrant-system"/"dns-operator-controller-manager-dockercfg-shlrr" (30-Oct-2026 20:09:34.863) (total time: 22016ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1326998532]: ---"Objects streamed" count:1 22016ms (20:09:56.879) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1326998532]: [22.016476455s] [22.016476455s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.879902 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-shlrr\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880010 5733 trace.go:236] Trace[1951678218]: "Reflector WatchList" name:object-"kuadrant-system"/"kuadrant-operator-controller-manager-dockercfg-kjn7n" (30-Oct-2026 20:09:34.944) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1951678218]: ---"Objects streamed" count:1 21935ms (20:09:56.879) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1951678218]: [21.93519661s] [21.93519661s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880017 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kjn7n\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880153 5733 trace.go:236] Trace[1078941888]: "Reflector WatchList" name:object-"openshift-cluster-csi-drivers"/"aws-ebs-csi-driver-node-sa-dockercfg-2cs46" (30-Oct-2026 20:09:34.891) (total time: 21988ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1078941888]: ---"Objects streamed" count:1 21988ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1078941888]: [21.988198621s] [21.988198621s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880162 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2cs46\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880301 5733 trace.go:236] Trace[1598914492]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w5-278ffa"/"distinct-2-w5-278ffa-kserve-self-signed-certs" (30-Oct-2026 20:09:34.892) (total time: 21987ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1598914492]: ---"Objects streamed" count:1 21987ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1598914492]: [21.987393946s] [21.987393946s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880309 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-278ffa\"/\"distinct-2-w5-278ffa-kserve-self-signed-certs\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880393 5733 trace.go:236] Trace[722771607]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-e2e-shared-b-w0-3d0a78-serving-cert" (30-Oct-2026 20:09:34.944) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[722771607]: ---"Objects streamed" count:1 21935ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[722771607]: [21.935490765s] [21.935490765s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880402 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-3d0a78-serving-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880454 5733 trace.go:236] Trace[1213038694]: "Reflector WatchList" name:object-"openshift-monitoring"/"alertmanager-main-web-config" (30-Oct-2026 20:09:34.945) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1213038694]: ---"Objects streamed" count:1 21935ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1213038694]: [21.935326895s] [21.935326895s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880459 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880501 5733 trace.go:236] Trace[447224680]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-rulefiles-2" (30-Oct-2026 20:09:34.945) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[447224680]: ---"Objects streamed" count:0 21935ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[447224680]: [21.935228811s] [21.935228811s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880506 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880579 5733 trace.go:236] Trace[1862577438]: "Reflector WatchList" name:object-"openshift-ingress-canary"/"canary-serving-cert" (30-Oct-2026 20:09:34.944) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1862577438]: ---"Objects streamed" count:1 21935ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1862577438]: [21.935608024s] [21.935608024s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880585 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880634 5733 trace.go:236] Trace[19766299]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-b-82885f" (30-Oct-2026 20:09:34.945) (total time: 21935ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[19766299]: ---"Objects streamed" count:1 21935ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[19766299]: [21.935280922s] [21.935280922s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880639 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-82885f\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880730 5733 trace.go:236] Trace[1507467900]: "Reflector WatchList" name:object-"openshift-ingress"/"kube-auth-proxy-dockercfg-fqw8g" (30-Oct-2026 20:09:34.893) (total time: 21987ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1507467900]: ---"Objects streamed" count:1 21987ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1507467900]: [21.987387175s] [21.987387175s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880736 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-fqw8g\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880791 5733 trace.go:236] Trace[583578542]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w1-a1d885"/"kube-root-ca.crt" (30-Oct-2026 20:09:34.906) (total time: 21974ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[583578542]: ---"Objects streamed" count:1 21974ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[583578542]: [21.974509953s] [21.974509953s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880796 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a1d885\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880855 5733 trace.go:236] Trace[837824277]: "Reflector WatchList" name:object-"llm"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.906) (total time: 21974ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[837824277]: ---"Objects streamed" count:1 21974ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[837824277]: [21.974495512s] [21.974495512s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880861 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880937 5733 trace.go:236] Trace[408279469]: "Reflector WatchList" name:object-"openshift-monitoring"/"node-exporter-tls" (30-Oct-2026 20:09:34.893) (total time: 21987ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[408279469]: ---"Objects streamed" count:1 21987ms (20:09:56.880) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[408279469]: [21.987260095s] [21.987260095s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.880944 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881010 5733 trace.go:236] Trace[1747816783]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-e2e-inf-a-a7bb6c-serving-cert" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1747816783]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1747816783]: [21.975773034s] [21.975773034s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881018 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a7bb6c-serving-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881120 5733 trace.go:236] Trace[1425234829]: "Reflector WatchList" name:object-"odh-ai-gateway-infra"/"maas-api-e2e-inf-a-4f244d-serving-cert" (30-Oct-2026 20:09:34.906) (total time: 21974ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1425234829]: ---"Objects streamed" count:1 21974ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1425234829]: [21.97468024s] [21.97468024s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881127 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-4f244d-serving-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881199 5733 trace.go:236] Trace[911163734]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-inf-a-1f246b-dockercfg-8chtl" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[911163734]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[911163734]: [21.975444884s] [21.975444884s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881208 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1f246b-dockercfg-8chtl\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881274 5733 trace.go:236] Trace[408415015]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"jobset-manager-config" (30-Oct-2026 20:09:34.906) (total time: 21974ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[408415015]: ---"Objects streamed" count:1 21974ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[408415015]: [21.974756952s] [21.974756952s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881281 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881358 5733 trace.go:236] Trace[1069530783]: "Reflector WatchList" name:object-"openshift-ovn-kubernetes"/"ovnkube-config" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1069530783]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1069530783]: [21.975878373s] [21.975878373s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881365 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881424 5733 trace.go:236] Trace[851809880]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-b-19f2ee" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[851809880]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[851809880]: [21.975592952s] [21.975592952s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881430 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-19f2ee\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881501 5733 trace.go:236] Trace[460708794]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-dockercfg-l4qwx" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[460708794]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[460708794]: [21.975908796s] [21.975908796s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881508 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4qwx\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881588 5733 trace.go:236] Trace[880703943]: "Reflector WatchList" name:object-"openshift-dns"/"dns-default-metrics-tls" (30-Oct-2026 20:09:34.893) (total time: 21987ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[880703943]: ---"Objects streamed" count:1 21987ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[880703943]: [21.987994123s] [21.987994123s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881594 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881682 5733 trace.go:236] Trace[338530061]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w2-d5e3e5"/"simulator-w2-d5e3e5-kserve-self-signed-certs" (30-Oct-2026 20:09:34.905) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[338530061]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[338530061]: [21.975735137s] [21.975735137s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881693 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-d5e3e5\"/\"simulator-w2-d5e3e5-kserve-self-signed-certs\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881809 5733 trace.go:236] Trace[1119745835]: "Reflector WatchList" name:object-"kube-system"/"global-pull-secret-syncer-dockercfg-2prmp" (30-Oct-2026 20:09:34.855) (total time: 22026ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1119745835]: ---"Objects streamed" count:1 22026ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1119745835]: [22.026392782s] [22.026392782s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881819 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-2prmp\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881949 5733 trace.go:236] Trace[622488240]: "Reflector WatchList" name:object-"openshift-multus"/"multus-ancillary-tools-dockercfg-wp4kx" (30-Oct-2026 20:09:34.906) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[622488240]: ---"Objects streamed" count:1 21975ms (20:09:56.881) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[622488240]: [21.975898835s] [21.975898835s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.881960 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wp4kx\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882092 5733 trace.go:236] Trace[880669193]: "Reflector WatchList" name:object-"openshift-ingress"/"istiod-openshift-gateway-dockercfg-sh8sg" (30-Oct-2026 20:09:34.906) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[880669193]: ---"Objects streamed" count:1 21975ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[880669193]: [21.975249067s] [21.975249067s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882105 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sh8sg\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882213 5733 trace.go:236] Trace[1331260664]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w4-3f57c5"/"kube-root-ca.crt" (30-Oct-2026 20:09:34.906) (total time: 21976ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1331260664]: ---"Objects streamed" count:1 21976ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1331260664]: [21.976041481s] [21.976041481s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882224 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882336 5733 trace.go:236] Trace[943262268]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w5-278ffa-dockercfg-cvh8s" (30-Oct-2026 20:09:34.906) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[943262268]: ---"Objects streamed" count:1 21975ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[943262268]: [21.975419801s] [21.975419801s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882342 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-278ffa-dockercfg-cvh8s\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882437 5733 trace.go:236] Trace[1565701822]: "Reflector WatchList" name:object-"openshift-network-operator"/"iptables-alerter-script" (30-Oct-2026 20:09:34.907) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1565701822]: ---"Objects streamed" count:1 21975ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1565701822]: [21.975253985s] [21.975253985s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882447 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882530 5733 trace.go:236] Trace[572469955]: "Reflector WatchList" name:object-"openshift-monitoring"/"serving-certs-ca-bundle" (30-Oct-2026 20:09:34.906) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[572469955]: ---"Objects streamed" count:1 21975ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[572469955]: [21.975533523s] [21.975533523s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882537 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882626 5733 trace.go:236] Trace[1072683182]: "Reflector WatchList" name:object-"openshift-image-registry"/"node-ca-dockercfg-xs4jl" (30-Oct-2026 20:09:34.907) (total time: 21975ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1072683182]: ---"Objects streamed" count:1 21975ms (20:09:56.882) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1072683182]: [21.975527515s] [21.975527515s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.882635 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xs4jl\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.883405 5733 trace.go:236] Trace[532734593]: "Reflector WatchList" name:object-"openshift-ingress-canary"/"ingress-canary-dockercfg-ttn9c" (30-Oct-2026 20:09:34.856) (total time: 22026ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[532734593]: ---"Objects streamed" count:1 22026ms (20:09:56.883) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[532734593]: [22.026498942s] [22.026498942s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:56.883421 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ttn9c\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.043495 5733 trace.go:236] Trace[1436070083]: "Reflector WatchList" name:object-"kuadrant-system"/"authorino-authorino-dockercfg-cq4cm" (30-Oct-2026 20:09:34.892) (total time: 22151ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1436070083]: ---"Objects streamed" count:1 22151ms (20:09:57.043) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1436070083]: [22.151155393s] [22.151155393s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.043519 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-cq4cm\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.045776 5733 trace.go:236] Trace[78499862]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w6-5f933e"/"default-dockercfg-dsn78" (30-Oct-2026 20:09:34.095) (total time: 22950ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[78499862]: ---"Objects streamed" count:1 22950ms (20:09:57.045) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[78499862]: [22.950470227s] [22.950470227s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.045794 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-5f933e\"/\"default-dockercfg-dsn78\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.045992 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.046184 5733 trace.go:236] Trace[370501968]: "Reflector WatchList" name:object-"openshift-monitoring"/"node-exporter-accelerators-collector-config" (30-Oct-2026 20:09:34.095) (total time: 22950ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[370501968]: ---"Objects streamed" count:1 22950ms (20:09:57.046) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[370501968]: [22.950805903s] [22.950805903s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.046195 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167283 5733 trace.go:236] Trace[415447860]: "Reflector WatchList" name:object-"openshift-monitoring"/"kube-state-metrics-kube-rbac-proxy-config" (30-Oct-2026 20:09:34.892) (total time: 22274ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[415447860]: ---"Objects streamed" count:1 22274ms (20:09:57.167) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[415447860]: [22.274519727s] [22.274519727s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167320 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167363 5733 trace.go:236] Trace[598019306]: "Reflector WatchList" name:object-"llm"/"e2e-trlp-test-simulated-kserve-self-signed-certs" (30-Oct-2026 20:09:34.947) (total time: 21848ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[598019306]: ---"Objects streamed" count:1 21848ms (20:09:56.796) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[598019306]: [21.848047438s] [21.848047438s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167371 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167876 5733 trace.go:236] Trace[860183885]: "Reflector WatchList" name:object-"openshift-jobset-operator"/"webhook-server-cert" (30-Oct-2026 20:09:34.855) (total time: 22312ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[860183885]: ---"Objects streamed" count:1 22312ms (20:09:57.167) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[860183885]: [22.31277272s] [22.31277272s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.167896 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.168230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.183969 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.184651 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.184889 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.184927 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.186362 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.80055361s: [/var/lib/containers/storage/overlay/275c9a879ac76be8d263647dad8d01cbbf2e413725902cd74e74e8b4e5428589/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.186402 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.757142854s: [/var/lib/containers/storage/overlay/4f4318be428a49a945d9769c6efa49fff8a5f8c0ebc3143b94d79babd1691e98/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.186432 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.748098612s: [/var/lib/containers/storage/overlay/ed252527448da4ea902c713c36d7c1aacf9422badf60d516337e00e307ad2b42/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.186795 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.700752527s: [/var/lib/containers/storage/overlay/f16b7c5e54c3e7db99446354c13be300b18585a9f9b5ef7b9a2909c439143da7/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.187716 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.190358 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.625057096s: [/var/lib/containers/storage/overlay/b0e64ff7ab40df5f0fc07f126be7937b38a55f62330a7642573ad617e62f163b/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal_e4925bc4d897d213bab4d8b38932653e/haproxy/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.190979 5733 trace.go:236] Trace[396177882]: "iptables ChainExists" (30-Oct-2026 20:09:40.244) (total time: 16946ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[396177882]: [16.946702032s] [16.946702032s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.191253 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.544238875s: [/var/lib/containers/storage/overlay/dc2841edc08d4e48d7a02ff013b6535321cfe299e24e7fb9ea347fbaad4b6f10/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.192273 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.507164993s: [/var/lib/containers/storage/overlay/308bb5534f69324cbf4475e294f5ccd4c747f2fd44b3ba16f26ed9b184368023/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.193873 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.456017918s: [/var/lib/containers/storage/overlay/68f42a823e90de6279e2431193e125c26b50ef95134aba41ef57e1e6aa743077/diff /var/log/pods/openshift-multus_network-metrics-daemon-8nnkd_33381540-cec9-4797-a8b9-be05dd1f13c0/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.193905 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.193928 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.205876 5733 trace.go:236] Trace[2090054475]: "Reflector WatchList" name:object-"kuadrant-system"/"kube-root-ca.crt" (30-Oct-2026 20:09:34.947) (total time: 22258ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[2090054475]: ---"Objects streamed" count:1 22258ms (20:09:57.205) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[2090054475]: [22.258368631s] [22.258368631s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.205890 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.205983 5733 trace.go:236] Trace[1738995555]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-tls" (30-Oct-2026 20:09:34.908) (total time: 22297ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1738995555]: ---"Objects streamed" count:1 22297ms (20:09:57.205) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1738995555]: [22.297790893s] [22.297790893s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.205991 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206048 5733 trace.go:236] Trace[1779337522]: "Reflector WatchList" name:object-"openshift-monitoring"/"node-exporter-kube-rbac-proxy-config" (30-Oct-2026 20:09:34.908) (total time: 22297ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1779337522]: ---"Objects streamed" count:1 22297ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1779337522]: [22.297779857s] [22.297779857s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206073 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206167 5733 trace.go:236] Trace[1244322116]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6" (30-Oct-2026 20:09:34.893) (total time: 22312ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1244322116]: ---"Objects streamed" count:1 22312ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1244322116]: [22.312740506s] [22.312740506s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206173 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8a8207-openshift-default-dockercfg-shmd6\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206252 5733 trace.go:236] Trace[421241787]: "Reflector WatchList" name:object-"openshift-multus"/"metrics-daemon-sa-dockercfg-8hwq2" (30-Oct-2026 20:09:35.803) (total time: 21403ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[421241787]: ---"Objects streamed" count:1 21403ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[421241787]: [21.403133829s] [21.403133829s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206258 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8hwq2\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206326 5733 trace.go:236] Trace[1883952921]: "Reflector WatchList" name:object-"e2e-models-e2e-worker-w4-3f57c5"/"default-dockercfg-6842t" (30-Oct-2026 20:09:34.893) (total time: 22312ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1883952921]: ---"Objects streamed" count:1 22312ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1883952921]: [22.312818391s] [22.312818391s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206333 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3f57c5\"/\"default-dockercfg-6842t\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206391 5733 trace.go:236] Trace[1661985635]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-shared-a-w3-274922" (30-Oct-2026 20:09:34.855) (total time: 22350ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1661985635]: ---"Objects streamed" count:1 22350ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1661985635]: [22.35090857s] [22.35090857s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206398 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-274922\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206492 5733 trace.go:236] Trace[207296451]: "Reflector WatchList" name:object-"openshift-ingress"/"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k" (30-Oct-2026 20:09:34.905) (total time: 22300ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[207296451]: ---"Objects streamed" count:1 22300ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[207296451]: [22.300819102s] [22.300819102s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206499 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206556 5733 trace.go:236] Trace[601842510]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-plugins-e2e-inf-a-925e04" (30-Oct-2026 20:09:34.905) (total time: 22301ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[601842510]: ---"Objects streamed" count:1 22301ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[601842510]: [22.301188115s] [22.301188115s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206561 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-925e04\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206943 5733 trace.go:236] Trace[1868269790]: "Reflector WatchList" name:object-"openshift-monitoring"/"prometheus-k8s-rulefiles-0" (30-Oct-2026 20:09:34.907) (total time: 22299ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1868269790]: ---"Objects streamed" count:1 22299ms (20:09:57.206) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1868269790]: [22.299654935s] [22.299654935s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.206954 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.207081 5733 trace.go:236] Trace[1689649362]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-worker-w2-d5e3e5-dockercfg-tbc7b" (30-Oct-2026 20:09:34.907) (total time: 22299ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1689649362]: ---"Objects streamed" count:1 22299ms (20:09:57.207) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1689649362]: [22.299718024s] [22.299718024s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.207088 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-d5e3e5-dockercfg-tbc7b\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.207155 5733 trace.go:236] Trace[1952516321]: "Reflector WatchList" name:object-"openshift-multus"/"default-cni-sysctl-allowlist" (30-Oct-2026 20:09:34.941) (total time: 22265ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1952516321]: ---"Objects streamed" count:1 22265ms (20:09:57.207) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1952516321]: [22.26540832s] [22.26540832s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.207162 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:57.215976 5733 logging.go:55] [core] [Channel #5600 SubChannel #5601]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.176:50052", ServerName: "10.133.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.176:50052: operation was canceled" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:09:57.219232 5733 logging.go:55] [core] [Channel #5602 SubChannel #5613]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.56:50052", ServerName: "10.133.0.56:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.56:50052: operation was canceled" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.244069 5733 ???:1] "http: TLS handshake error from 10.133.0.16:53020: EOF" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.253654 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.177:50052\" within 1s: context deadline exceeded" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.342661 5733 trace.go:236] Trace[1248381770]: "Reflector WatchList" name:object-"openshift-ingress"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.941) (total time: 22401ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1248381770]: ---"Objects streamed" count:1 22401ms (20:09:57.342) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1248381770]: [22.401011868s] [22.401011868s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.342680 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.352337 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a7bb6c-dockercfg-5z5xs\": awaiting required bookmark event for initial events stream, no events received for 15.609307699s" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.352401 5733 trace.go:236] Trace[955662988]: "Reflector WatchList" name:object-"openshift-ingress"/"payload-processing-e2e-inf-a-a7bb6c-dockercfg-5z5xs" (30-Oct-2026 20:09:34.094) (total time: 23257ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[955662988]: ---"Objects streamed" count:1 23257ms (20:09:57.352) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[955662988]: [23.257997791s] [23.257997791s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.352411 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a7bb6c-dockercfg-5z5xs\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.407998 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 16.680365194s" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.408122 5733 trace.go:236] Trace[1424493182]: "Reflector WatchList" name:object-"openshift-ovn-kubernetes"/"openshift-service-ca.crt" (30-Oct-2026 20:09:33.989) (total time: 23418ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1424493182]: ---"Objects streamed" count:1 23418ms (20:09:57.408) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1424493182]: [23.418834134s] [23.418834134s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.408135 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.408159 5733 reflector.go:1159] "Warning: event bookmark expired" err="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\": awaiting required bookmark event for initial events stream, no events received for 15.805563729s" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.408198 5733 trace.go:236] Trace[1381502742]: "Reflector WatchList" name:object-"openshift-cluster-node-tuning-operator"/"openshift-service-ca.crt" (30-Oct-2026 20:09:34.864) (total time: 22544ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1381502742]: ---"Objects streamed" count:1 22544ms (20:09:57.408) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[1381502742]: [22.544164157s] [22.544164157s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.408205 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.448989 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-ngwls" podUID="63e2b306-7302-41be-92fa-0c0dbeb0e6be" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.133.0.33:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.480339 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.57:50052\" within 1s: context deadline exceeded" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.498989 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.499296 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-nrp79" podUID="72fe1029-ecc8-4b99-9387-659ae6e414b3" containerName="extractor" probeResult="failure" output="command timed out" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.499319 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.499374 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.704966 5733 trace.go:236] Trace[272273667]: "Reflector WatchList" name:pkg/kubelet/config/apiserver.go:66 (30-Oct-2026 20:09:34.864) (total time: 22840ms): Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[272273667]: ---"Objects streamed" count:120 22840ms (20:09:57.704) Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: Trace[272273667]: [22.840394624s] [22.840394624s] END Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.704987 5733 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.883667 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:57.924166 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.045761 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Oct 30 20:10:23.991710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.084347 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.086178 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.086217 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.187477 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.217173 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.217234 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.219320 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.223568 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.233256 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.266142 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.270165 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.270212 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.270949 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.270991 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.271782 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.271828 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.275104 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.275136 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.276188 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.276224 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.285160 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.285219 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.388185 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.388233 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.389216 5733 patch_prober.go:29] interesting pod/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.389252 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.390219 5733 patch_prober.go:29] interesting pod/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.390259 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.405857 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.434225 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.434287 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.447173 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-zj7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.447213 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.451282 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.455192 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.455229 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.473202 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.473242 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.482175 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.482207 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.524172 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.524215 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.526181 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.526210 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.531167 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.531196 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:58.858729 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.086986 5733 patch_prober.go:29] interesting pod/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.087033 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.179374 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.265404 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.270871 5733 patch_prober.go:29] interesting pod/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.270911 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.271956 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.271999 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.272138 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.272185 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.276143 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.276172 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.277204 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.277234 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.307187 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.456959 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.836157 5733 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-ng7cn container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.836202 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:09:59.930373 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1m43.888s" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:09:59.951499 5733 generic.go:356] "Generic (PLEG): container finished" podID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerID="3e71065b395cfface84d2648874d38eb412a42167e2ebc5fdef122321694ef0e" exitCode=1 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.628788 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.628966 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": context deadline exceeded" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.650205 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="cert-manager-webhook" containerStatusID={"Type":"cri-o","ID":"0b4442e6aa1a879be1b2fb00cc6bb681c4c200ce506718e1a3e063cdea0966e1"} pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" containerMessage="Container cert-manager-webhook failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.650258 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" containerID="cri-o://0b4442e6aa1a879be1b2fb00cc6bb681c4c200ce506718e1a3e063cdea0966e1" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.651130 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.651175 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" containerID="cri-o://282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.651739 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="metrics-server" containerStatusID={"Type":"cri-o","ID":"e91cda68bca3bac20fe06da4495966a7f7b1e4b1c44993f8a6c43a40a9825f63"} pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" containerMessage="Container metrics-server failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.651779 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" containerID="cri-o://e91cda68bca3bac20fe06da4495966a7f7b1e4b1c44993f8a6c43a40a9825f63" gracePeriod=170 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.651982 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.652016 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" containerID="cri-o://3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.652378 5733 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" secret="" err="secret \"payload-processing-e2e-inf-a-925e04-dockercfg-tstq2\" not found" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.659297 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"8f6f1a7ab236f09929bdcaeca86bf70d85d8bedf8c2b54a400acc867f408a7e7"} pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.659340 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" containerID="cri-o://8f6f1a7ab236f09929bdcaeca86bf70d85d8bedf8c2b54a400acc867f408a7e7" gracePeriod=10 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.664606 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": context deadline exceeded" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.666203 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.670110 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"6e17da26fcad1398b2c5f91ba8739150e073bf7233d06fb2c1ca47d7dcbecbff"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" containerMessage="Container csi-driver failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.670142 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"87d043921fbc201018fa8815a227531edc0fb6e7d424d11c84b9b8016baef344"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.670185 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-driver" containerID="cri-o://6e17da26fcad1398b2c5f91ba8739150e073bf7233d06fb2c1ca47d7dcbecbff" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.707445 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.707476 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.744115 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"b2a69ffb4d46878be72aef5733be04704c160667c3017bb4249856965774fc45"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.744160 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" containerID="cri-o://b2a69ffb4d46878be72aef5733be04704c160667c3017bb4249856965774fc45" gracePeriod=10 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.809247 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1"} pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.809355 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" containerID="cri-o://ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1" gracePeriod=10 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.810822 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.810872 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" containerID="cri-o://4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.811183 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"3e71065b395cfface84d2648874d38eb412a42167e2ebc5fdef122321694ef0e"} pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.811227 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" containerID="cri-o://3e71065b395cfface84d2648874d38eb412a42167e2ebc5fdef122321694ef0e" gracePeriod=10 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.813585 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="dns" containerStatusID={"Type":"cri-o","ID":"e45e18d2836cd828e4c5a4186ac9075fb789ff0d6b902e9be8f887ab9bede091"} pod="openshift-dns/dns-default-bjt85" containerMessage="Container dns failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.813675 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" containerID="cri-o://e45e18d2836cd828e4c5a4186ac9075fb789ff0d6b902e9be8f887ab9bede091" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.840593 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d"} pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.840682 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" containerID="cri-o://8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d" gracePeriod=30 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:00.842235 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:00.842303 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:01.342273564 +0000 UTC m=+2361.410109112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.943574 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:00.954019 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:00.954122 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:01.454102648 +0000 UTC m=+2361.521938194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:00.977176 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:01.368003 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:01.368086 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:02.368067195 +0000 UTC m=+2362.435902755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:01.471289 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:01.471356 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:02.471337597 +0000 UTC m=+2362.539173143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.610274 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6h66z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.610335 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:01.885008 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.228s" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885090 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885123 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885167 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" event={"ID":"03223b13-85fc-4dda-bf3b-dec632a94c83","Type":"ContainerDied","Data":"3e71065b395cfface84d2648874d38eb412a42167e2ebc5fdef122321694ef0e"} Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885321 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885422 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885467 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885489 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885537 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885561 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885614 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.885641 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886096 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886156 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886179 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886203 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886224 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886250 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886276 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886300 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886823 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886849 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.886906 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.887384 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.887418 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.887809 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.887931 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.888300 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.888755 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.890474 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.890916 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.891612 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896597 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896666 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896718 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896739 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896765 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896790 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896816 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896841 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896866 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896895 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896928 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896967 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.896991 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897008 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897029 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897079 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897106 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897130 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897181 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 20:10:24.640834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897207 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897235 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897260 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kms8d" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897282 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897298 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897333 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897351 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897375 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897391 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897444 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897471 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897519 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897559 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897576 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897622 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897660 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897688 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897704 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897731 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897757 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897781 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897806 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897831 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897857 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897882 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897905 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897929 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897949 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897971 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.897995 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898019 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898083 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898109 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898136 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898160 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898186 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898211 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898235 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898260 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898286 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898334 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898378 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898441 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.898521 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.899623 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.903284 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.906397 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.907374 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.907982 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.946796 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.946914 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.946991 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947216 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947255 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947405 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947463 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947611 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.947808 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.951078 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.953926 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955751 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955803 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955833 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955855 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955881 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-6slx7" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955898 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955934 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955965 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.955996 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.956026 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.956052 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.956122 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.956154 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.956186 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:01.957259 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.097362 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.097824 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.097890 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098775 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098837 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098870 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098904 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098942 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.098973 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.099392 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.102047 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.102678 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.102738 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.102778 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.179838 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.184701 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.279275 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-kms8d" podUID="46dad1ef-96d8-483e-909b-c082e47acbbd" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.365243 5733 patch_prober.go:29] interesting pod/metrics-server-7bd8695446-nbt67 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:02.399159 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:02.501328 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.930172 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.932149 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.951255 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:02.956149 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.073624 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494296 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494458 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494507 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" podUID="38afb684-e999-4c10-9281-62f633431cc6" containerName="metrics-server" probeResult="failure" output="Get \"https://10.133.0.13:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494534 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:03.494545 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:05.494519539 +0000 UTC m=+2365.562355104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494569 5733 ???:1] "http: TLS handshake error from 10.133.0.13:56680: read tcp 10.0.139.103:10250->10.133.0.13:56680: read: connection reset by peer" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:03.494641 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:05.494621767 +0000 UTC m=+2365.562457317 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494644 5733 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.494658 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.495155 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" containerMessage="Container maas-api failed liveness probe, will be restarted" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.495198 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" containerID="cri-o://860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307" gracePeriod=30 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.629369 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.629634 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.664948 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:03.667471 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.161249 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.301272 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.301364 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.301397 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.301505 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.375942 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kms8d" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.536168 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.536211 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.536167 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:04.536167 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:06.702790 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:06.715695 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:06.734258 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:06.734473 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:10.734453076 +0000 UTC m=+2370.802288628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:06.735116 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:06.735170 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:10.735156808 +0000 UTC m=+2370.802992360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:06.766076 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:08.095967 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:08.100541 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:08.100625 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:08.100672 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:09.263239 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:09.813089 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" containerID="cri-o://69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98" gracePeriod=24 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:10.657159 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.629394 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.633442 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" containerID="cri-o://02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9" gracePeriod=22 Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.636376 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.640438 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.664482 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.664828 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.665131 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.665413 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:11.665699 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:12.655851 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:12.655922 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:12.656200 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:12.812893 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:12.813018 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:20.812986705 +0000 UTC m=+2380.880822268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:12.813093 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:12.813125 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:20.813113821 +0000 UTC m=+2380.880949367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:13.073049 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:13.650973 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" podUID="0c7a23be-1c57-4b4f-a494-0ff740d58fca" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.737714 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.740823 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.740864 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.740925 5733 patch_prober.go:29] interesting pod/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.740940 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.740978 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.741255 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.59:8443/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.741347 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.742433 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.743161 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.743267 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.743346 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.743465 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.744812 5733 patch_prober.go:29] interesting pod/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.744861 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.744931 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.747562 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.747898 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.119:50052\" within 1s: context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.748601 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.748740 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.749132 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.57:50052: i/o timeout" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.750643 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.750795 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.750835 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.751146 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": context deadline exceeded" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.751626 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.868116021s: [/var/lib/containers/storage/overlay/2beae6ad824d2ea25277f19427753fe664c51d68b1390c55ef1fd60f37e7d54e/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:10.659449 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.753941 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.070772 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.107862 5733 ???:1] "http: TLS handshake error from 10.133.0.16:53030: read tcp 10.0.139.103:10250->10.133.0.16:53030: read: connection reset by peer" Oct 30 20:10:24.794243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.714768 5733 patch_prober.go:29] interesting pod/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.714840 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.909632 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.910820 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.923642 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.924021 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.754844 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.755288 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.171:50052: i/o timeout" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.755304 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.755366 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.755608 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.755929 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.756146 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.756444 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.757924 5733 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6h66z container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.757957 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.761925 5733 trace.go:236] Trace[770258992]: "Calculate volume metrics of tmp-dir for pod e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" (30-Oct-2026 20:10:11.392) (total time: 5375ms): Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: Trace[770258992]: [5.375197507s] [5.375197507s] END Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:17.572806 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.58:50052: i/o timeout" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.764577 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.169048423s: [/var/lib/containers/storage/overlay/f827e5f8aa0140b85715e474bd1c49ce46e6813b8c1a5ffb023070398e252e9c/diff ]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:18.764609 5733 logging.go:55] [core] [Channel #5787 SubChannel #5788]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.119:50052", ServerName: "10.133.0.119:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.119:50052: operation was canceled" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.781714 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.781762 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.795095 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.796446 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": dial tcp 10.133.0.32:6080: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.852995 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.856221 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.856354 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.51:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.856518 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.856803 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" podUID="294d869b-a6be-48ff-95f3-8123552c5d4f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:18.864510 5733 logging.go:55] [core] [Channel #5800 SubChannel #5823]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.170:50052: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.919872 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.919967 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.943038 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" podUID="cd58d3a0-2332-494d-85f0-e48f263ac29c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.959189 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": dial tcp 10.133.0.35:8081: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:18.966935 5733 ???:1] "http: TLS handshake error from 10.133.0.16:53036: read tcp 10.0.139.103:10250->10.133.0.16:53036: read: connection reset by peer" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.017089 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.114:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.026682 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.031393 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.033172 5733 logging.go:55] [core] [Channel #5789 SubChannel #5841]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.114:50052", ServerName: "10.133.0.114:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.114:50052: operation was canceled" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.053302 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": dial tcp 10.133.0.29:8081: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.078233 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": dial tcp 10.133.0.27:8081: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.079347 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-nrp79" podUID="72fe1029-ecc8-4b99-9387-659ae6e414b3" containerName="extractor" probeResult="failure" output="command timed out" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.081657 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.56:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.081836 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.081868 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.081881 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.081904 5733 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-103.us-east-2.compute.internal" event="NodeReady" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.089312 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": dial tcp 10.133.0.162:8443: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.090739 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.091086 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ef38f10e-731b-49df-a5c8-0fce38cc59ab" containerName="prometheus" probeResult="failure" output="command timed out" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.091538 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": dial tcp 10.133.0.162:8443: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.103705 5733 logging.go:55] [core] [Channel #5856 SubChannel #5857]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.169:50052", ServerName: "10.133.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.169:50052: connect: connection refused" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:16.743117 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.104145 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" podUID="952eb089-1681-4eac-94d4-9f7465666913" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.118309 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.128378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.131042 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.960387349s: [/var/lib/containers/storage/overlay/6362475accbc9c6701c585fa522b2349e2d737e809a583886ca4bdab330e8368/diff /var/log/pods/openshift-ingress_e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq_90effab3-a893-40cd-b620-673575f1d679/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.131105 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.475129789s: [/var/lib/containers/storage/overlay/24dfa8e7e93dd5849fd0e2f0d88636f205220464800d88b5fa833e2900323308/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovn-controller/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.131127 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.47515088s: [/var/lib/containers/storage/overlay/33ab0a87f689f1542b5e07726c4ec2196ad1236851e11025239b84b2fea10b4c/diff /var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.134625 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.480800893s: [/var/lib/containers/storage/overlay/93230bf5ce6af9cf80071a11874a3e77f65cfd971cffe170da69ba1b8beb01e5/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137353 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.483312703s: [/var/lib/containers/storage/overlay/58046d9378d92d0692fb638f1283465480e12dfda98c4117b990f1404a4dd793/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/nbdb/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137710 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.483040198s: [/var/lib/containers/storage/overlay/e01fb8fb6bc2017ad854a78689b2d7e29983c5e8b5aa463c5011912a97725ee4/diff /var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/node-exporter/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137728 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482489056s: [/var/lib/containers/storage/overlay/e2db25e4caa5ca969ebdf08215bc1903512941931099b226082c40ec09556a27/diff /var/log/pods/openshift-multus_multus-2hcjh_eed84a66-cd93-4e58-adb3-e097e1b17a14/kube-multus/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137740 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 10.920131201s: [/var/lib/containers/storage/overlay/e8b937bb70c28849a3c976c525d3e884bff89fff6ae659ddeae4261f95b689d0/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc_8ed984b5-27dc-47f6-b3a3-0be88a09763e/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137760 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.486385412s: [/var/lib/containers/storage/overlay/0f72861fc0b556d4d3c70373be8f55e20d0d421faa24cf77e4c8cf918e8f5980/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-v484k_99d64cc3-7d77-4972-8df4-4f7040a5de6f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137772 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481985153s: [/var/lib/containers/storage/overlay/b537639028490d06b9fb4abbcbe9d1b2f4e07e822ed9f99c547a16f6fcaede86/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-rcvnc_6049f1ce-d07c-4f37-9a3c-dfeff0232cf2/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137783 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.486834664s: [/var/lib/containers/storage/overlay/c651e6200300fc1fb3affd693c64d443df50afdc453b514e1c471bc8c7021f98/diff /var/log/pods/openshift-ingress_e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2_4aae276e-c30b-44e3-af8d-d88aa479cfb4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137794 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482419943s: [/var/lib/containers/storage/overlay/2cfe011ebd9ae6a1368270084c7dc8aec4d527b27da2f2a292312f80f2603c43/diff /var/log/pods/e2e-models-e2e-worker-w5-278ffa_distinct-2-w5-278ffa-kserve-c69ccf475-jppkh_d142cf77-146e-4719-8e79-001e99aead3c/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137805 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.485067991s: [/var/lib/containers/storage/overlay/52e99bca613012bc41a5e4bb074e09d798c559880b435f8b0d6e23c9c2f3f2dc/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-gdnn8_7489a135-99a1-41d0-8b09-1288c18ecf8b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137843 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.386790518s: [/var/lib/containers/storage/overlay/4d2f2e228b30a3a6606dcd5c9596f3351a753943ffe37ebe7068b9e363948a5e/diff ]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137861 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48367802s: [/var/lib/containers/storage/overlay/2435fc1d58b871cbe00e52576c94da4dc0967b551f4065915a98878fbd71557c/diff /var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137877 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.485310794s: [/var/lib/containers/storage/overlay/6fb31e9813f119ca30f25fb22b211b2f302c1a41f4cd3b79d894f435e386824b/diff /var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-dwm4z_7a8bac1e-c263-46f7-97e7-860f85904afc/jobset-operator/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137892 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.484027642s: [/var/lib/containers/storage/overlay/41c9003eb4ce7e3dfe83bc3ff3602685a94a50887a62fe47a8feb550e8b0a37d/diff /var/log/pods/e2e-models-e2e-worker-w1-a1d885_distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr_d550cab4-61b4-43c5-ae10-4eb9597cc7b0/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137904 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.484932709s: [/var/lib/containers/storage/overlay/e91d4725d482d270bf943d948f0980f1cc0afc87c66c219c99cef9cfcace1354/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc_3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137916 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481938035s: [/var/lib/containers/storage/overlay/3a0e2994a5fc77a8fe59aeec232bc6848c10393371a17887d3e833caa76bb930/diff /var/log/pods/openshift-ingress_e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h_bdd45e19-244b-44e1-87c0-d18e32f1de37/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137927 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481969586s: [/var/lib/containers/storage/overlay/9bd1ba2ff6f6c78ba7cbe586e5dd36bfa5d175366453d2076cb76f4936f3dad4/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-82885f-55777b478d-khbl8_382161a6-f53d-4795-b7c4-005dd3e1ca94/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137938 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482243011s: [/var/lib/containers/storage/overlay/1768924c685e4b87d23b289d11779209411086521fc36df7515c3a614b769d35/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj_51bc38b8-b8b2-434e-819b-e7502f4d4441/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137950 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482772609s: [/var/lib/containers/storage/overlay/797aee25ebb624bc0831e870ecb1320ff691dc45227d4b6a2763be947fcab7c9/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/extractor/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137960 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481949727s: [/var/lib/containers/storage/overlay/9528dfa863d8f54c349e8abeb5199826cacca04ca44bff318f2114da1a6788f5/diff /var/log/pods/openshift-multus_network-metrics-daemon-8nnkd_33381540-cec9-4797-a8b9-be05dd1f13c0/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137972 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481941178s: [/var/lib/containers/storage/overlay/04edfee9b803991c4f7fa145ca5cefba7c4f1a3e5c9a53868ab1299cc371ab65/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137982 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481948451s: [/var/lib/containers/storage/overlay/16f16e38d632c83f931f5f75f399bf18015371cdf86a673a2fd6912d363182b3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.137993 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481957136s: [/var/lib/containers/storage/overlay/9145540bdea9a3c96ee30241738486c54106f1aae15487e0220f7d86de5a3697/diff /var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138003 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.4819653s: [/var/lib/containers/storage/overlay/ab80a0d7a25f78c788c581eb45019ffa3f9791cb8e0c4d3e9d7ace42ffc72215/diff /var/log/pods/e2e-models-e2e-worker-w6-5f933e_simulator-w6-5f933e-kserve-86bf55fd5b-pncbq_5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138038 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481997955s: [/var/lib/containers/storage/overlay/e1d263946dd0c4ca5e584378dcb1f8c445544d364b1b0ea9f5e8ae13194c9aa7/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x_47b19494-6b48-41e6-956a-713a34b0dc10/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138050 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.486403369s: [/var/lib/containers/storage/overlay/a465a08fa16c83236d866ed0885cbe7f4e36417588e10b43a13a9f94142ec59c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx_8b964b0d-e172-433f-b5cb-7fd19823d2d7/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138077 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482095716s: [/var/lib/containers/storage/overlay/3e25f6b5818c9d7664c64585c3c114ec4044f7efc9c8b0943f5a5dbf256b3d7f/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s_c9306a89-0fc2-44cd-9aa2-96f5ec782c02/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138090 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48204796s: [/var/lib/containers/storage/overlay/aa21c62792bc33953c2cd21dfed1409b8ec18ea5ad994d996e6ae07451c4e17c/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_distinct-w2-d5e3e5-kserve-c67646f86-96xsp_4e877fbf-b484-4cfc-93a9-b2e813a364b2/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138101 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482128296s: [/var/lib/containers/storage/overlay/38d72313e754a571559ed480e33d982353b0a900a0cad70033e7ff24c66afe6f/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88_058c7110-65ab-4311-9688-b389e30c5f2a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138113 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481578139s: [/var/lib/containers/storage/overlay/541d816d7209c577f8fd0be27f4ed51adc18ca9f3b2e0b1b712d599fe33f5941/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138123 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481587293s: [/var/lib/containers/storage/overlay/3805159434d64aff1ea46c7646d7f4497eef42c0ddb63ef15e7786f0b9f14a66/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138133 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481595829s: [/var/lib/containers/storage/overlay/c23bd4b52006b5789aac48a6cc6900b02f5d87c5d6db911c533c5b4859882e3b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/northd/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138146 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482162882s: [/var/lib/containers/storage/overlay/e1df21269b56c680209b681bf062d852c2e53652c59c03c61c7957148c732b86/diff /var/log/pods/openshift-ingress_e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f_a74bb1f9-6fa8-4a48-92ef-62b50d195958/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138157 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48217229s: [/var/lib/containers/storage/overlay/b63b5cb8797ad5c2e17e0847ca2cfb5840391e56a38b66f455073066355a8ce9/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-ngwls_63e2b306-7302-41be-92fa-0c0dbeb0e6be/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138167 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481623353s: [/var/lib/containers/storage/overlay/342d363d463ab87d01f69cca23c0a75b6b0603ac5c33dc80c2bde12419f5e368/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138178 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481632492s: [/var/lib/containers/storage/overlay/c7e154b51cea889b993c9b5d372cf163100c9296f3c1ce940d92987bd35f851a/diff /var/log/pods/openshift-network-diagnostics_network-check-target-mrpl8_ce385369-c46a-4bdd-800a-8ac5ae39a471/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138188 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481641219s: [/var/lib/containers/storage/overlay/014f1563fa52d38b34cb2ac176fd125a18c8fca4a87f7b740b6361c80cbbd9f5/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138200 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48165091s: [/var/lib/containers/storage/overlay/693fcbfdd0bda2b572dd905f87386213d64fbe8bb41506afc01c0da23a006b35/diff /var/log/pods/openshift-ingress_e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6_e7520762-fb61-4512-875b-f376e9a2bbda/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138211 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48165992s: [/var/lib/containers/storage/overlay/319054daf94c26aa0b7c451fb41d5fee89f0fb34516561a7de3f8d956d2a296c/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138222 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481668461s: [/var/lib/containers/storage/overlay/2bf5394c4070c85769f116a50902a4f201b5469d39c7eb6d03ed7a0463a874a3/diff /var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-state-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138232 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482185786s: [/var/lib/containers/storage/overlay/0252af765c74679864c17d877aa751eefe6cf9ddbad09c835e899bf31dfaf599/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt_bd91d8c7-be08-4975-b46b-70f110ca1bfc/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138243 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481673559s: [/var/lib/containers/storage/overlay/16d4d0bd740886dc2781d247a1fc5758b93f80bfbd933e0f4fc2e384776aa6ee/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138254 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481682623s: [/var/lib/containers/storage/overlay/220e1468aed6d6dfdc06972a2f87a1cf70fc1b71f6079b0ca14a6f4375816683/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-2qlwk_e8b76e60-de88-4c65-b570-fb4772dfbb5b/tuned/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138266 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481692976s: [/var/lib/containers/storage/overlay/7d5cbafe7194b9430fcfd420ef2ada65830a1fb01b4edfc399250d38937b0a2c/diff /var/log/pods/openshift-ingress_e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h_b9558d5f-e925-43dc-90b4-928cd373c624/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138276 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481736473s: [/var/lib/containers/storage/overlay/a1107cd7fd3cd031a34dee21ad2e363c4f865333fc846656b9d69dd6604d2dc6/diff /var/log/pods/openshift-dns_node-resolver-jcmd8_ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138286 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482243008s: [/var/lib/containers/storage/overlay/74ed5f850d45ab14b404c50727f92e4030e076a9331ce671fe884d37e9f34527/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w_cb994d3f-ac88-4fff-b9ee-400f5556cafb/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138297 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482317234s: [/var/lib/containers/storage/overlay/8907b2a2a94770d20f8d510ea95d231076d18e5d9a3c36347bd1e3ae7053f043/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz_d05d05bb-294b-4d00-a60f-98fd0d84bfdb/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138309 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482261105s: [/var/lib/containers/storage/overlay/f5464b00f60f0c82c89ff3bc406ab74753755b99093e5adab31a7827647b06e1/diff /var/log/pods/cert-manager_cert-manager-webhook-5dbbbf75cc-glbct_3bb9e4d1-26a1-4698-9322-d963463664b6/cert-manager-webhook/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138320 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481739746s: [/var/lib/containers/storage/overlay/3038d194b3f4dd8c5b2e67ce1ee77388f7f3b38b021420e835f86c657526ff18/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw_f4c14d49-8d28-428b-80a5-51a881aae5d1/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138331 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481749152s: [/var/lib/containers/storage/overlay/25db7643fb80eee942d42689d53de0d585e52cfbf827d0e3fde717003d74ed3e/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-85768f955d-hxzvg_82727e18-39c9-438c-b335-259eb7281109/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138342 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481800744s: [/var/lib/containers/storage/overlay/7ea9f2b1e18efc8821d0d2c51fcd33383de7b22dc1102828d6f13ea5a310a5d4/diff /var/log/pods/openshift-ingress_e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4_b535075b-1639-47bb-b428-7c5c0eb5d4b2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138352 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481775129s: [/var/lib/containers/storage/overlay/bf9a97cd93828c5ea7cc3928d0c262510b94e278f3e08d6a39845f157473c463/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr_b700fa9e-7b5a-42f7-9d54-551c147eccc8/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138363 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481775528s: [/var/lib/containers/storage/overlay/bd5a9efc23d281b533ec11e2f8680cf233516f3bcde0f03afc43648812c50e18/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/config-reloader/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138374 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481778593s: [/var/lib/containers/storage/overlay/d32ce8e422358cc390acf5ca696b8fddeed34d92f19e7715552410d135e03227/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138386 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481830775s: [/var/lib/containers/storage/overlay/a3d5d84b16196139f840cbc2b7b7ca0b1db7a17301ea101856aad25039f55028/diff /var/log/pods/openshift-ingress_e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc_ff87fcd9-988d-4f08-aaf3-86d2ff50bab6/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138396 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481797866s: [/var/lib/containers/storage/overlay/29b4535f1c1b12b757a66a681741d590f25cb786680a561069ed7d3737cbb3b2/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138407 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481830895s: [/var/lib/containers/storage/overlay/5e77f67c43a70bdb70a015274a69b587442d9643ee63c6331f2c1c627e5b1d64/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc_7733aeff-8a16-47ab-ae96-b3ee56b1ef38/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138417 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481816345s: [/var/lib/containers/storage/overlay/3341736d23442d6c635b476020f9cdb98a5e8a8ab81a302b72e266bdc17d87ef/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138428 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481849277s: [/var/lib/containers/storage/overlay/0f743dc2092fcd2e2efb8238a68face705b584d286c440fb589453036ea82b09/diff /var/log/pods/openshift-ingress_e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb_f117faf1-f824-4851-9ce6-e85ebd8fa690/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138440 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481854515s: [/var/lib/containers/storage/overlay/4cd6bfa788cc712734ec643ce2791749c621439d9de857cb636c5bafafe02233/diff /var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138450 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481866495s: [/var/lib/containers/storage/overlay/7d44de1f35baed0cb48ff040891238122c3f70632540a9431adc71730b329e88/diff /var/log/pods/e2e-models-e2e-worker-w4-3f57c5_premium-w4-3f57c5-kserve-c49958c8f-85rhc_86e76d74-7775-4b7b-9120-885c993ad626/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138461 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481858695s: [/var/lib/containers/storage/overlay/0b0574bedc52d86a95c4c57bbc5027c65f482876a71a690fbdc83acad5d52a77/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/prometheus/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138471 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481871381s: [/var/lib/containers/storage/overlay/4ac363fa4deec1786013cd643c91c774e6259217479c2c183a220b47a769a243/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138482 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481875327s: [/var/lib/containers/storage/overlay/a49c683f30142faf7b36fde60beacfefb8f403a5ee75449b1b055a371e951796/diff /var/log/pods/openshift-ingress_e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj_f34c469c-95a6-4134-b889-d20cbe59c1a3/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138493 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481918546s: [/var/lib/containers/storage/overlay/1d885477065e41f9f74b33eff686c3c1bda03e20e1b357c1c8cead2e150515e5/diff /var/log/pods/e2e-models-e2e-worker-w4-3f57c5_distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4_64f65e7a-479c-44cd-9f40-5aea55deb3e1/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138504 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481881728s: [/var/lib/containers/storage/overlay/1c7eb20bbb70896a39bcb506f47da51482bce5214b06e89264e7cd9826be7d08/diff /var/log/pods/openshift-ingress_e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm_b78a1582-b7a4-4742-a25a-94209cae3c42/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138515 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481918088s: [/var/lib/containers/storage/overlay/90e8881dbf612fd21e0940270c75cb53ed04f1a5255bedaf181c45ae57025973/diff /var/log/pods/openshift-ingress_e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg_51892042-d2ae-4a69-9cc4-e347399249ed/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138526 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481901618s: [/var/lib/containers/storage/overlay/e5640c3037d3a35c2c5af767ec9b9fd268a6154260c7ac39adae8fd4a3220c8d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl_6862aa9e-2401-4f55-b63d-25435bf3eac4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138536 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482550237s: [/var/lib/containers/storage/overlay/0535997866c26a64d78f4aecf2211c32465b6ea7763785688725dbe8ed21ce7d/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138546 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482559344s: [/var/lib/containers/storage/overlay/ce0546fdc06499cd220770cd25c8004e4411b263f866bb7fb0243c179af7f0e5/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6kbw_b36aff52-af76-477f-990d-9900e4c43954/sbdb/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.034369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138558 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481925838s: [/var/lib/containers/storage/overlay/9f9bda789fdb612df4515b72d1054cee40659fb1abe5c64f1fdeaa63b955dbcb/diff /var/log/pods/e2e-models-e2e-worker-w6-5f933e_distinct-w6-5f933e-kserve-594b74cb-9fzpf_d1c5d55c-b95f-4c74-9bbb-4d660b088cfa/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138569 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482581048s: [/var/lib/containers/storage/overlay/d443fdcb8b6dd63985648f5576839be11b38b50ce827557df31af41b59d563a3/diff /var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/exporter/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138580 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482583089s: [/var/lib/containers/storage/overlay/028e24f7f2a4cb3bea8c03fd6ea0f7819bffd2cd7ebab78992de71ed7a1e6846/diff /var/log/pods/openshift-network-operator_iptables-alerter-2pmgc_29bace5a-fb35-45e9-a2ce-41a61af06524/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138590 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481986858s: [/var/lib/containers/storage/overlay/cf40e9a6bd7307229527715a77b3d0e3c20df991e5a8983e670506ef000b7a94/diff /var/log/pods/openshift-image-registry_node-ca-7gh2g_ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/node-ca/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138601 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.481995526s: [/var/lib/containers/storage/overlay/e225e7fca28dd374498776f82c1158c224640ae62d2045166894ef70a91c24dc/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg_1fe5eb2e-3136-4690-815f-ead77c1a5384/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138613 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482004229s: [/var/lib/containers/storage/overlay/dbe40c0dff2605fd9a14492d700bd0cb1e74c41c31c76276582f03375e726a42/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t_3433ab80-4a84-4013-80d2-7c7555c470d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138624 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482013666s: [/var/lib/containers/storage/overlay/f6746051b4cda06bab23c025632f16d0c078477d15979796d2966c0a804dd244/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7_f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138636 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482007731s: [/var/lib/containers/storage/overlay/aa052981fee07675cd25a9d170a1d4ead36c63862700cb514dddb2c8228a3096/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk_33c8e7e1-6494-49a8-b61c-bb3098dcccde/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138647 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482042527s: [/var/lib/containers/storage/overlay/061d7fb9b35a55d54f9f6521de6492034a63c0c26e925fb26e5898372f1f9010/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb_ddb50a54-7c85-49dc-abfd-d9234707efd1/maas-api/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138657 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482021375s: [/var/lib/containers/storage/overlay/f1b3bc27cd01da413aea250f0076183d604116e5da53cf1cb6d5151010165db1/diff /var/log/pods/e2e-models-e2e-worker-w3-8a8207_distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8_f3c61292-cd14-4230-80ac-2aaf2d55be7c/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138669 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.48203836s: [/var/lib/containers/storage/overlay/1a17084cbc53035e47a17e1c8cf6fa5483d1b41f48c241e8cbbbd914cc59d8dd/diff /var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138681 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.482054509s: [/var/lib/containers/storage/overlay/d5288b19a987ece6283670a42ff6d62407915c57a164598878595c014746d51f/diff /var/log/pods/e2e-models-e2e-worker-w3-8a8207_distinct-w3-8a8207-kserve-68f4987c5d-zbc97_11fec734-033b-44bb-a7a4-5a3f81ecddeb/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138692 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.505753526s: [/var/lib/containers/storage/overlay/3e036d721aabdc25130ba677f76e63429b4dd1fa95d522d56d482464ca80894d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq_4b80ef9e-12fc-48a1-a912-891358436d57/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138702 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.326429548s: [/var/lib/containers/storage/overlay/9276feab5386d24697101985b69449a1565e2a5ecec88b52db47a5da33217db9/diff /var/log/pods/openshift-ingress_e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6_9db2cd8e-f002-40ae-9ca1-66041fdec5bd/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.138714 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.505779765s: [/var/lib/containers/storage/overlay/ce0b5893616fb0df5977f6ef75efdf6c1da67df33482416d3728f19265f389a5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm_dc31087c-8882-44ff-9240-42cf43576885/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.139892 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.483251682s: [/var/lib/containers/storage/overlay/fab476cc812599913bd5bc4517f7761c07f02cf432b2c7e676b48c2cec355b78/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl_229609ef-e005-41c3-807a-54fa69b874d9/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.140240 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.483595212s: [/var/lib/containers/storage/overlay/d853ce0f7c8afbc838f0028a269b259ae3d026954ff8217512782e702f3926a1/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt_78000019-d14f-4c89-b1d4-06dfcbd4be4e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.140359 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.483717577s: [/var/lib/containers/storage/overlay/385e8a9c5efa91bdb0807303db453221d27970777f413aac4ee43d28ab60cb34/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m_5c087e21-e692-4eb2-8347-baa42ddc21d3/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.140685 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.484682172s: [/var/lib/containers/storage/overlay/809e505e260ebe8b625bffd3a3b63b579377ecb8322f490f7e01df9eca93c746/diff /var/log/pods/openshift-monitoring_metrics-server-7bd8695446-nbt67_38afb684-e999-4c10-9281-62f633431cc6/metrics-server/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.140999 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.484996486s: [/var/lib/containers/storage/overlay/1b120743020a1387ad4db47ccb5a61d7b2434e892e971cf29c3f2d47a480d74c/diff /var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/config-reloader/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.142642 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.486658245s: [/var/lib/containers/storage/overlay/ecd4d1b146cdabf99864fce66465ab78de5befe508499592022f70e42a684d0e/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-l89fd_651c14dc-17ea-4608-b927-ba1af3ea8b41/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.144386 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.526614678s: [/var/lib/containers/storage/overlay/ae2b70151fb30b6bfca874819236dd1c2cc84fff5dde2c91218f575383d74dfb/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-zj7hb_35a6c5c6-c04a-40b6-a97a-828c0bd2d86c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.145728 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.512781265s: [/var/lib/containers/storage/overlay/f5bc013bf9e786123ac615380d98a862fb6322480a06df414b30e017f26dc5d8/diff /var/log/pods/openshift-ingress_e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs_52995638-b72e-460b-8306-f442c213ff19/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.149677 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.531899142s: [/var/lib/containers/storage/overlay/df4f6394762baff1144b9d7a7b628d355717bee896eec46c71f01fbe7bdcb37f/diff /var/log/pods/e2e-models-e2e-worker-w2-d5e3e5_premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr_30e3bfe8-a903-4119-b1a6-46417f98910f/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.151299 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.077104299s: [/var/lib/containers/storage/overlay/028e444c8b45a0b21f26608e574cf36bcafd21641f18b49769a098bd7c4e1f00/diff /var/log/pods/openshift-ingress-canary_ingress-canary-bct4d_0678842b-b564-4b68-ae5b-aaffd1680479/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.151747 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.533969924s: [/var/lib/containers/storage/overlay/1bb4f092e5ad015e2a2111ea77279894bd40e9f4d4d581e6690f60b5a627c37d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v_66363c5c-5f40-47f5-bb62-4b00184dba74/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.153586 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.520645815s: [/var/lib/containers/storage/overlay/fa19b90cab0806d9f426f3bb9d923a291e151f7040aee2258623aa68e7beee3d/diff /var/log/pods/openshift-ingress_e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr_56d233c6-d1fc-4b59-8998-afe61144f065/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.167252 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.166:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.168659 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.168990 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.54:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.169168 5733 logging.go:55] [core] [Channel #5791 SubChannel #5861]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.169436 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.164:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.172592 5733 logging.go:55] [core] [Channel #5792 SubChannel #5862]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.54:50052", ServerName: "10.133.0.54:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.54:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.174420 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.52:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.174619 5733 logging.go:55] [core] [Channel #5793 SubChannel #5863]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.164:50052", ServerName: "10.133.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.164:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.174836 5733 logging.go:55] [core] [Channel #5790 SubChannel #5860]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.166:50052", ServerName: "10.133.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.166:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.199239 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.159:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.199759 5733 logging.go:55] [core] [Channel #5796 SubChannel #5865]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.159:50052", ServerName: "10.133.0.159:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.159:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.200338 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.207291 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.221405 5733 logging.go:55] [core] [Channel #5794 SubChannel #5864]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.52:50052", ServerName: "10.133.0.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.52:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.247393 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.254646 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.262679 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.292185 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.63617018s: [/var/lib/containers/storage/overlay/e72402eb7c1275e12d2717e37c4836feed966dd897ced9988b246554c4f1a0b1/diff /var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/dns/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.309329 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.653317905s: [/var/lib/containers/storage/overlay/baafef32837f7c0b4418e56028b46214876b7b9af1a90ec35e5e662d13fe5506/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-6h66z_9a7e5628-1d20-497b-a673-d28a54b125f2/discovery/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.310789 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.654776179s: [/var/lib/containers/storage/overlay/721d79f09217f76171f5e68c5a6f55aa038f691a785c7efa10f893c8a2f38172/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq_0c7a23be-1c57-4b4f-a494-0ff740d58fca/main/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.310871 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.654862185s: [/var/lib/containers/storage/overlay/ff5ab19ad73cca11b947316277eaa03e0cdd50660953830aa31aa84859695795/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx_88acb5d2-4a06-4b9f-a408-0137dad64956/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.311867 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.655857264s: [/var/lib/containers/storage/overlay/31024a7533bdef4a43370a4cf9e77247a4b656d471ebbb1dda045c4c8aaf7337/diff /var/log/pods/openshift-ingress_e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq_8b1adab2-ae8a-4b80-9273-97fb0d04e5ac/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.312171 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.656162687s: [/var/lib/containers/storage/overlay/237b0cd74bb66d72ec74ceb25a86750dfe4012d38e02e43ca31702b70d79a52e/diff /var/log/pods/openshift-ingress_e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw_4b1744f9-9ed1-4e7b-a359-8a1499160126/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.333130 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-ngwls" podUID="63e2b306-7302-41be-92fa-0c0dbeb0e6be" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.133.0.33:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.333466 5733 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 8.677449949s: [/var/lib/containers/storage/overlay/f49add07ffcd788d6cc26b4d6914193eeea91d10bb273ee9d4c7516efb27ee51/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 3s Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.357706 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.172:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:19.357984 5733 logging.go:55] [core] [Channel #5797 SubChannel #5867]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.172:50052", ServerName: "10.133.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.172:50052: operation was canceled" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.361092 5733 generic.go:356] "Generic (PLEG): container finished" podID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerID="8f6f1a7ab236f09929bdcaeca86bf70d85d8bedf8c2b54a400acc867f408a7e7" exitCode=137 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.366417 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:19.371295 5733 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="11.175s" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.414022 5733 generic.go:356] "Generic (PLEG): container finished" podID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerID="0b4442e6aa1a879be1b2fb00cc6bb681c4c200ce506718e1a3e063cdea0966e1" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.422827 5733 generic.go:356] "Generic (PLEG): container finished" podID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerID="282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.425701 5733 generic.go:356] "Generic (PLEG): container finished" podID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerID="860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.446793 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:19.519793 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.592880 5733 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.624860 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.624975 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" event={"ID":"56747b0a-051b-4e92-a51a-b71b394a2de2","Type":"ContainerDied","Data":"8f6f1a7ab236f09929bdcaeca86bf70d85d8bedf8c2b54a400acc867f408a7e7"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.625014 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642016 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642078 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" event={"ID":"3bb9e4d1-26a1-4698-9322-d963463664b6","Type":"ContainerDied","Data":"0b4442e6aa1a879be1b2fb00cc6bb681c4c200ce506718e1a3e063cdea0966e1"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642106 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerDied","Data":"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642138 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerDied","Data":"860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642166 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642708 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642751 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642788 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642817 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642843 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642869 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.642969 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.643447 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.643491 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.643525 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.643944 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.644312 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.644426 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.644513 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.644827 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" containerID="cri-o://971c24d0caa4087ea0b18f3e1a5fed49f146c79ec5060061df0705ba51390b26" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.644964 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645022 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645084 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645116 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645182 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645224 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645255 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645289 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645330 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645362 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645397 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645432 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-tn875" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645481 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645514 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-nrlrs" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645553 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645587 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645634 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645677 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645712 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-q9sb8" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645760 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645806 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.645896 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.648024 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" containerID="cri-o://6f5d15c9cde2e1e9e132f5ef785511cd6041e47025ae39e2b4f7ea2b4a9790bc" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.648847 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" containerID="cri-o://99161c544933d962fe7ffd9827eced0782c37807ef4eb3118699eb7dfb3af1a1" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.649022 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" containerID="cri-o://e98008f879ac4beff479e8c78adfd03b818726f79887cd84a36feb340a6d2684" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.649158 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" containerID="cri-o://dab333a75bc732f29a57f9f02819c0a628d87bdec8733309ce2d719cd08572b8" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.659923 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" containerID="cri-o://4c4761934862cdc7a3f6b2cc0d1c5b9799955b1bb161babf36a63c9a37329ce6" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.660079 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" containerID="cri-o://107db65fd094c0b5f82e71abc806daeab1926735e830876b97614a7a65397855" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.660474 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" containerID="cri-o://432546abde812b81a38c559ac44aedba642da56220811edccaeeb6b39c49d08d" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.660911 5733 generic.go:356] "Generic (PLEG): container finished" podID="82727e18-39c9-438c-b335-259eb7281109" containerID="b2a69ffb4d46878be72aef5733be04704c160667c3017bb4249856965774fc45" exitCode=137 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.661048 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerDied","Data":"b2a69ffb4d46878be72aef5733be04704c160667c3017bb4249856965774fc45"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.661344 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" containerID="cri-o://df3346d191c5ea28010f0a63c829ddc9995698d9f92f0081cf32b9f2acd89225" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.661691 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" containerID="cri-o://f1f15dca5e73b7c55271a553ff58fdc66d89066c623b6b4086ada73b2da21a60" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:19.687580 5733 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3433ab80_4a84_4013_80d2_7c7555c470d3.slice/crio-conmon-69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb994d3f_ac88_4fff_b9ee_400f5556cafb.slice/crio-conmon-02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:19.687832 5733 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3433ab80_4a84_4013_80d2_7c7555c470d3.slice/crio-conmon-69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:19.691492 5733 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb994d3f_ac88_4fff_b9ee_400f5556cafb.slice/crio-02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3433ab80_4a84_4013_80d2_7c7555c470d3.slice/crio-conmon-69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.701899 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" podUID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerName="csi-node-driver-registrar" containerID="cri-o://87d043921fbc201018fa8815a227531edc0fb6e7d424d11c84b9b8016baef344" gracePeriod=30 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.703254 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.769631 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.170:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.796573 5733 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.799390 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.799539 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.796373 5733 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"867d0dc3-f715-4fba-ae34-e967286bb78f","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:19.936004 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" podUID="82727e18-39c9-438c-b335-259eb7281109" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.27:8081/readyz\": dial tcp 10.133.0.27:8081: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.094895 5733 generic.go:356] "Generic (PLEG): container finished" podID="21b6051c-d265-4cd2-b08b-b64054880109" containerID="3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.095100 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerDied","Data":"3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.099566 5733 generic.go:356] "Generic (PLEG): container finished" podID="47b19494-6b48-41e6-956a-713a34b0dc10" containerID="4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.099599 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerDied","Data":"4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.102095 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.169:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.105564 5733 generic.go:356] "Generic (PLEG): container finished" podID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerID="8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.105639 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerDied","Data":"8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.107322 5733 generic.go:356] "Generic (PLEG): container finished" podID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerID="ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1" exitCode=137 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.107407 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerDied","Data":"ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.299760 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.299801 5733 generic.go:356] "Generic (PLEG): container finished" podID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerID="6e17da26fcad1398b2c5f91ba8739150e073bf7233d06fb2c1ca47d7dcbecbff" exitCode=2 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.299864 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerDied","Data":"6e17da26fcad1398b2c5f91ba8739150e073bf7233d06fb2c1ca47d7dcbecbff"} Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.478308 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:20.570550 5733 logging.go:55] [core] [Channel #5871 SubChannel #5872]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.170:50052: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:20.570661 5733 logging.go:55] [core] [Channel #5873 SubChannel #5874]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.169:50052", ServerName: "10.133.0.169:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.169:50052: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.630011 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.667273 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.708425 5733 patch_prober.go:29] interesting pod/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:20.708477 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:20.912109 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:20.912195 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume podName:cb994d3f-ac88-4fff-b9ee-400f5556cafb nodeName:}" failed. No retries permitted until 2026-10-30 20:10:36.912176718 +0000 UTC m=+2396.980012274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:20.912239 5733 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-925e04: configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:20.912274 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume podName:3433ab80-4a84-4013-80d2-7c7555c470d3 nodeName:}" failed. No retries permitted until 2026-10-30 20:10:36.912263271 +0000 UTC m=+2396.980098824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume") pod "payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3") : configmap "payload-processing-plugins-e2e-inf-a-925e04" not found Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.253589 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.359553 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv"] Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.367148 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-gdnn8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.367202 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.384280 5733 generic.go:356] "Generic (PLEG): container finished" podID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerID="02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.387794 5733 generic.go:356] "Generic (PLEG): container finished" podID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerID="69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.414021 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.414082 5733 generic.go:356] "Generic (PLEG): container finished" podID="d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5" containerID="87d043921fbc201018fa8815a227531edc0fb6e7d424d11c84b9b8016baef344" exitCode=0 Oct 30 20:10:25.097232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.418927 5733 trace.go:236] Trace[1026564067]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" (30-Oct-2026 20:09:06.404) (total time: 75014ms): Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: Trace[1026564067]: [1m15.014549883s] [1m15.014549883s] END Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.572005 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.169:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.572077 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.170:50052\" within 1s: context deadline exceeded" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:21.572207 5733 logging.go:55] [core] [Channel #5871 SubChannel #5872]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.170:50052: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.647710 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" event={"ID":"cb994d3f-ac88-4fff-b9ee-400f5556cafb","Type":"ContainerDied","Data":"02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9"} Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.647772 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" event={"ID":"3433ab80-4a84-4013-80d2-7c7555c470d3","Type":"ContainerDied","Data":"69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98"} Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.647799 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerDied","Data":"87d043921fbc201018fa8815a227531edc0fb6e7d424d11c84b9b8016baef344"} Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.647971 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.654648 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.654982 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.707901 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.707954 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.771624 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.772108 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.772214 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9klwp\" (UniqueName: \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.873813 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.874185 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.874250 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9klwp\" (UniqueName: \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.910542 5733 patch_prober.go:29] interesting pod/jobset-controller-manager-64d7d576f9-fb9qh container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" start-of-body= Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.910593 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" podUID="03223b13-85fc-4dda-bf3b-dec632a94c83" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.34:8081/readyz\": dial tcp 10.133.0.34:8081: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:21.910857 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": dial tcp 10.133.0.35:8081: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:22.391500 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:22.589544 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.059710 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv"] Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.146756 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.495811 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" podUID="3bb9e4d1-26a1-4698-9322-d963463664b6" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.32:6080/healthz\": dial tcp 10.133.0.32:6080: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.495868 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" podUID="56747b0a-051b-4e92-a51a-b71b394a2de2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.29:8081/readyz\": dial tcp 10.133.0.29:8081: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.563533 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.630439 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.666154 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:23.902230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.091440 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.162:8443/health\": dial tcp 10.133.0.162:8443: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.434845 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.694605 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.708976 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.709120 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:24.954963 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.197147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.196865 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.428960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.380610 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9klwp\" (UniqueName: \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\") pod \"maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.466477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.461850 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:25.473416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.471471 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" event={"ID":"cb994d3f-ac88-4fff-b9ee-400f5556cafb","Type":"ContainerDied","Data":"52a1c191c33d86cc4d2fdd81fb1affb6e062fa174e1c776b21726dd6becaf840"} Oct 30 20:10:25.473416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.471551 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52a1c191c33d86cc4d2fdd81fb1affb6e062fa174e1c776b21726dd6becaf840" Oct 30 20:10:25.473416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.472937 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" event={"ID":"3433ab80-4a84-4013-80d2-7c7555c470d3","Type":"ContainerDied","Data":"0dfe2d739c1c44b8936879da9e7ca75d2c4b2e76214f576ca277e6a305cab2c8"} Oct 30 20:10:25.473416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.472975 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dfe2d739c1c44b8936879da9e7ca75d2c4b2e76214f576ca277e6a305cab2c8" Oct 30 20:10:25.504306 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.504267 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:10:25.510161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.510114 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:10:25.583578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.582132 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\") pod \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " Oct 30 20:10:25.583578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.582377 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\" (UniqueName: \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\") pod \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\" (UID: \"cb994d3f-ac88-4fff-b9ee-400f5556cafb\") " Oct 30 20:10:25.583578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.582417 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\" (UniqueName: \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\") pod \"3433ab80-4a84-4013-80d2-7c7555c470d3\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " Oct 30 20:10:25.583578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.582481 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\") pod \"3433ab80-4a84-4013-80d2-7c7555c470d3\" (UID: \"3433ab80-4a84-4013-80d2-7c7555c470d3\") " Oct 30 20:10:25.627921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.627882 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume" pod "3433ab80-4a84-4013-80d2-7c7555c470d3" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:25.630148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.630022 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.118:8443/health\": dial tcp 10.133.0.118:8443: connect: connection refused" Oct 30 20:10:25.667335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.667301 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.161:8443/health\": dial tcp 10.133.0.161:8443: connect: connection refused" Oct 30 20:10:25.685322 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.685295 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3433ab80-4a84-4013-80d2-7c7555c470d3-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:25.697235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.696977 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume" pod "cb994d3f-ac88-4fff-b9ee-400f5556cafb" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:25.807330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.774495 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.807330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.788069 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb994d3f-ac88-4fff-b9ee-400f5556cafb-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:25.817775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.817584 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8" pod "3433ab80-4a84-4013-80d2-7c7555c470d3" (UID: "3433ab80-4a84-4013-80d2-7c7555c470d3"). InnerVolumeSpecName "kube-api-access-lswj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:25.817775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.817649 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn" pod "cb994d3f-ac88-4fff-b9ee-400f5556cafb" (UID: "cb994d3f-ac88-4fff-b9ee-400f5556cafb"). InnerVolumeSpecName "kube-api-access-5pjzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:25.894084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.892234 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pjzn\" (UniqueName: \"kubernetes.io/projected/cb994d3f-ac88-4fff-b9ee-400f5556cafb-kube-api-access-5pjzn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:25.894084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:25.892283 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lswj8\" (UniqueName: \"kubernetes.io/projected/3433ab80-4a84-4013-80d2-7c7555c470d3-kube-api-access-lswj8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:26.480021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.479835 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:26.491141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.490885 5733 generic.go:356] "Generic (PLEG): container finished" podID="90effab3-a893-40cd-b620-673575f1d679" containerID="107db65fd094c0b5f82e71abc806daeab1926735e830876b97614a7a65397855" exitCode=0 Oct 30 20:10:26.491663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.491431 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerDied","Data":"107db65fd094c0b5f82e71abc806daeab1926735e830876b97614a7a65397855"} Oct 30 20:10:26.491663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.491531 5733 scope.go:122] "RemoveContainer" containerID="04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f" Oct 30 20:10:26.495094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.494970 5733 generic.go:356] "Generic (PLEG): container finished" podID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerID="e45e18d2836cd828e4c5a4186ac9075fb789ff0d6b902e9be8f887ab9bede091" exitCode=0 Oct 30 20:10:26.495176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.495092 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjt85" event={"ID":"b107b211-0176-4170-8d0f-85d4b36a4f30","Type":"ContainerDied","Data":"e45e18d2836cd828e4c5a4186ac9075fb789ff0d6b902e9be8f887ab9bede091"} Oct 30 20:10:26.499534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.499506 5733 generic.go:356] "Generic (PLEG): container finished" podID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerID="6f5d15c9cde2e1e9e132f5ef785511cd6041e47025ae39e2b4f7ea2b4a9790bc" exitCode=0 Oct 30 20:10:26.499640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.499587 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" event={"ID":"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c","Type":"ContainerDied","Data":"6f5d15c9cde2e1e9e132f5ef785511cd6041e47025ae39e2b4f7ea2b4a9790bc"} Oct 30 20:10:26.501994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.501974 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" event={"ID":"03223b13-85fc-4dda-bf3b-dec632a94c83","Type":"ContainerStarted","Data":"b5cba44f3c089d6e2914892682bd41647b55d9eb63610800c9315f0d1785ffcd"} Oct 30 20:10:26.502437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.502420 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 20:10:26.504884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.504862 5733 generic.go:356] "Generic (PLEG): container finished" podID="51892042-d2ae-4a69-9cc4-e347399249ed" containerID="971c24d0caa4087ea0b18f3e1a5fed49f146c79ec5060061df0705ba51390b26" exitCode=0 Oct 30 20:10:26.504983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.504942 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" event={"ID":"51892042-d2ae-4a69-9cc4-e347399249ed","Type":"ContainerDied","Data":"971c24d0caa4087ea0b18f3e1a5fed49f146c79ec5060061df0705ba51390b26"} Oct 30 20:10:26.506782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.506762 5733 generic.go:356] "Generic (PLEG): container finished" podID="b9558d5f-e925-43dc-90b4-928cd373c624" containerID="f1f15dca5e73b7c55271a553ff58fdc66d89066c623b6b4086ada73b2da21a60" exitCode=0 Oct 30 20:10:26.506869 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.506838 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" event={"ID":"b9558d5f-e925-43dc-90b4-928cd373c624","Type":"ContainerDied","Data":"f1f15dca5e73b7c55271a553ff58fdc66d89066c623b6b4086ada73b2da21a60"} Oct 30 20:10:26.508523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.508502 5733 generic.go:356] "Generic (PLEG): container finished" podID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerID="4c4761934862cdc7a3f6b2cc0d1c5b9799955b1bb161babf36a63c9a37329ce6" exitCode=0 Oct 30 20:10:26.508611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.508587 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" event={"ID":"f5652aad-c74e-4732-8bd7-1d814d64be92","Type":"ContainerDied","Data":"4c4761934862cdc7a3f6b2cc0d1c5b9799955b1bb161babf36a63c9a37329ce6"} Oct 30 20:10:26.510581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.510558 5733 generic.go:356] "Generic (PLEG): container finished" podID="52995638-b72e-460b-8306-f442c213ff19" containerID="df3346d191c5ea28010f0a63c829ddc9995698d9f92f0081cf32b9f2acd89225" exitCode=0 Oct 30 20:10:26.510660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.510637 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" event={"ID":"52995638-b72e-460b-8306-f442c213ff19","Type":"ContainerDied","Data":"df3346d191c5ea28010f0a63c829ddc9995698d9f92f0081cf32b9f2acd89225"} Oct 30 20:10:26.512718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.512694 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerStarted","Data":"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6"} Oct 30 20:10:26.512820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.512805 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:10:26.515038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.515020 5733 generic.go:356] "Generic (PLEG): container finished" podID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerID="432546abde812b81a38c559ac44aedba642da56220811edccaeeb6b39c49d08d" exitCode=0 Oct 30 20:10:26.515153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.515102 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" event={"ID":"4b1744f9-9ed1-4e7b-a359-8a1499160126","Type":"ContainerDied","Data":"432546abde812b81a38c559ac44aedba642da56220811edccaeeb6b39c49d08d"} Oct 30 20:10:26.516939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.516918 5733 generic.go:356] "Generic (PLEG): container finished" podID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerID="e98008f879ac4beff479e8c78adfd03b818726f79887cd84a36feb340a6d2684" exitCode=0 Oct 30 20:10:26.517044 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.516996 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" event={"ID":"4aae276e-c30b-44e3-af8d-d88aa479cfb4","Type":"ContainerDied","Data":"e98008f879ac4beff479e8c78adfd03b818726f79887cd84a36feb340a6d2684"} Oct 30 20:10:26.561348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.561316 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:10:26.585267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.585160 5733 generic.go:356] "Generic (PLEG): container finished" podID="56d233c6-d1fc-4b59-8998-afe61144f065" containerID="99161c544933d962fe7ffd9827eced0782c37807ef4eb3118699eb7dfb3af1a1" exitCode=0 Oct 30 20:10:26.585484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.585322 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerDied","Data":"99161c544933d962fe7ffd9827eced0782c37807ef4eb3118699eb7dfb3af1a1"} Oct 30 20:10:26.596881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.596846 5733 generic.go:356] "Generic (PLEG): container finished" podID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerID="dab333a75bc732f29a57f9f02819c0a628d87bdec8733309ce2d719cd08572b8" exitCode=0 Oct 30 20:10:26.597146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.596960 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" event={"ID":"7489a135-99a1-41d0-8b09-1288c18ecf8b","Type":"ContainerDied","Data":"dab333a75bc732f29a57f9f02819c0a628d87bdec8733309ce2d719cd08572b8"} Oct 30 20:10:26.600823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.600789 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" event={"ID":"56747b0a-051b-4e92-a51a-b71b394a2de2","Type":"ContainerStarted","Data":"4da3021493a2ec71233805253cd309bb8020a62e03847c52cc2594e300bec4e6"} Oct 30 20:10:26.601204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.601187 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 20:10:26.603655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.603635 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t" Oct 30 20:10:26.604265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.604239 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" event={"ID":"3bb9e4d1-26a1-4698-9322-d963463664b6","Type":"ContainerStarted","Data":"f2f26b2decb2b8762da572d75d0eacec18d69e90839562ecc2af80569145d821"} Oct 30 20:10:26.604611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.604591 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w" Oct 30 20:10:26.604817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.604602 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 20:10:26.671900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.671813 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:10:26.681437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.681405 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-925e04-84959f4df9-4628w"] Oct 30 20:10:26.708494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.708464 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:26.746159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.746119 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:10:26.840189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.840117 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-925e04-76496944cd-n5f9t"] Oct 30 20:10:26.863042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:26.863002 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 20:10:27.010507 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.010473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:27.036711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.036682 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.036881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.036750 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.036881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.036853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.036989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.036940 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.037094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.037011 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.037158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.037118 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.037244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.037230 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.037322 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.037307 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.037388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.037372 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\") pod \"f5652aad-c74e-4732-8bd7-1d814d64be92\" (UID: \"f5652aad-c74e-4732-8bd7-1d814d64be92\") " Oct 30 20:10:27.097160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.097131 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 20:10:27.107937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.107909 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.140534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.140509 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.143474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.143430 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.181851 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.181709 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.188676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.188286 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "kube-api-access-qdcxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.213659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.189162 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.213659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.192075 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.213659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.195179 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.213659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.195593 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 20:10:27.237412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.237371 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.244885 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245038 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245091 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245136 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245210 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245349 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245489 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245520 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.245624 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\") pod \"b9558d5f-e925-43dc-90b4-928cd373c624\" (UID: \"b9558d5f-e925-43dc-90b4-928cd373c624\") " Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246590 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246618 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246634 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246651 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdcxc\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-kube-api-access-qdcxc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246666 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246681 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.246998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.246698 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5652aad-c74e-4732-8bd7-1d814d64be92-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.247938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.247612 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy" pod "f5652aad-c74e-4732-8bd7-1d814d64be92" (UID: "f5652aad-c74e-4732-8bd7-1d814d64be92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.248250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.248108 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:27.259390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.259272 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 20:10:27.263434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.263213 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.263755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.263709 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.280699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.280635 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.284734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.284189 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.287139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.287019 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.302274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.302215 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.304241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.304188 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.311394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.311363 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.338622 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.348287 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.350876 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.350928 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351085 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351125 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351162 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351300 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351345 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351392 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351449 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351503 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351528 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351553 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351570 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351591 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351617 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\") pod \"4b1744f9-9ed1-4e7b-a359-8a1499160126\" (UID: \"4b1744f9-9ed1-4e7b-a359-8a1499160126\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351701 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351727 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.351802 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\") pod \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\" (UID: \"4aae276e-c30b-44e3-af8d-d88aa479cfb4\") " Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353391 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353418 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353433 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353448 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353463 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5652aad-c74e-4732-8bd7-1d814d64be92-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353476 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9558d5f-e925-43dc-90b4-928cd373c624-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353492 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.358475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.353506 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9558d5f-e925-43dc-90b4-928cd373c624-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.373586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.359095 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.373586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.363278 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.373586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.369166 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.376625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.376584 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.376926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.376889 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht" pod "b9558d5f-e925-43dc-90b4-928cd373c624" (UID: "b9558d5f-e925-43dc-90b4-928cd373c624"). InnerVolumeSpecName "kube-api-access-ltvht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.379240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.379049 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 20:10:27.386944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.386916 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.387049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.386959 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.387579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.387558 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.388571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.388547 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.390578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.390535 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.408182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.408156 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 20:10:27.425005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.424955 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.425176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.425004 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "kube-api-access-z8lnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.429859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.428036 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.429859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.428402 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.429859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.428683 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.429859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.428904 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert" pod "4b1744f9-9ed1-4e7b-a359-8a1499160126" (UID: "4b1744f9-9ed1-4e7b-a359-8a1499160126"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.430584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.430564 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "kube-api-access-g7bdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.430756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.430719 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.430918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.430891 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.431352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.431331 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token" pod "4aae276e-c30b-44e3-af8d-d88aa479cfb4" (UID: "4aae276e-c30b-44e3-af8d-d88aa479cfb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.453931 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455430 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455500 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455545 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455583 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455671 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455766 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455795 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455874 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455933 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.455963 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456014 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456045 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456190 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456301 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456405 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456502 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456611 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.456679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456685 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456715 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456768 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456843 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.456965 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.457086 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.457225 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\") pod \"56d233c6-d1fc-4b59-8998-afe61144f065\" (UID: \"56d233c6-d1fc-4b59-8998-afe61144f065\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.457266 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.457348 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\") pod \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\" (UID: \"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.457392 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\") pod \"52995638-b72e-460b-8306-f442c213ff19\" (UID: \"52995638-b72e-460b-8306-f442c213ff19\") " Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458624 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458656 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b1744f9-9ed1-4e7b-a359-8a1499160126-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458672 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8lnz\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-kube-api-access-z8lnz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458688 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458701 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458720 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458733 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9558d5f-e925-43dc-90b4-928cd373c624-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458746 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458762 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458781 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458793 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458807 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458820 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458833 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458847 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7bdx\" (UniqueName: \"kubernetes.io/projected/4aae276e-c30b-44e3-af8d-d88aa479cfb4-kube-api-access-g7bdx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458862 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b1744f9-9ed1-4e7b-a359-8a1499160126-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458881 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458895 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458909 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aae276e-c30b-44e3-af8d-d88aa479cfb4-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.462042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.458922 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvht\" (UniqueName: \"kubernetes.io/projected/b9558d5f-e925-43dc-90b4-928cd373c624-kube-api-access-ltvht\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.535911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.494677 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.535911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.517565 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:27.535911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.533226 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559589 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559698 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559807 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559838 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559901 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559920 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.559958 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560005 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560034 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560112 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560174 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560240 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560296 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560327 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560355 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560410 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560446 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560525 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560573 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560634 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560724 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560754 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560892 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\") pod \"51892042-d2ae-4a69-9cc4-e347399249ed\" (UID: \"51892042-d2ae-4a69-9cc4-e347399249ed\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.560975 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\") pod \"7489a135-99a1-41d0-8b09-1288c18ecf8b\" (UID: \"7489a135-99a1-41d0-8b09-1288c18ecf8b\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.561289 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\") pod \"90effab3-a893-40cd-b620-673575f1d679\" (UID: \"90effab3-a893-40cd-b620-673575f1d679\") " Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.562468 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.579862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.562487 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.602311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.580136 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.602311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.587720 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.602311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.597808 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "kube-api-access-cv6bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.608672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.608566 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.615378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.614389 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "kube-api-access-x6b7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.617514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.616535 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.617514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.616988 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.617514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.617360 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.626174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.626151 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" event={"ID":"4aae276e-c30b-44e3-af8d-d88aa479cfb4","Type":"ContainerDied","Data":"c72da464285303ada699e44d9c5e7ffdb6e41af5ceb6693a8f6e12d75e36027e"} Oct 30 20:10:27.626331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.626277 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2" Oct 30 20:10:27.633045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.633016 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" event={"ID":"56d233c6-d1fc-4b59-8998-afe61144f065","Type":"ContainerDied","Data":"2e1a8ef944300868f3eefcb491d2a4c9aaffa01dd37bb96820aced91596d3b79"} Oct 30 20:10:27.633468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.633047 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.634805 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" event={"ID":"7489a135-99a1-41d0-8b09-1288c18ecf8b","Type":"ContainerDied","Data":"1e617e7b8601a82567128068ac4b561ea60a6a3710dccfaa359f9198965ed10f"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.634877 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.637310 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" event={"ID":"90effab3-a893-40cd-b620-673575f1d679","Type":"ContainerDied","Data":"77f017ec4cc1a98ddc94f57bc7660fdde23e7f8bf9b4355fa0fd544aea65dd8a"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.637377 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.638982 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.638997 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb" event={"ID":"35a6c5c6-c04a-40b6-a97a-828c0bd2d86c","Type":"ContainerDied","Data":"a38c0bd5c6e4132a3396c3f425d998c0bbd0baf8383eb9a44cf0befb9a9853af"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.640547 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" event={"ID":"51892042-d2ae-4a69-9cc4-e347399249ed","Type":"ContainerDied","Data":"a57538f1222e3b32814363675a0747dc15b655e4234743478f7dab48ce935eab"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.640625 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.641925 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.642149 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.642173 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h" event={"ID":"b9558d5f-e925-43dc-90b4-928cd373c624","Type":"ContainerDied","Data":"ecb143c105945f3d38646672f5fb831637cd512e835b33a6fe1edbde532e65a2"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.643787 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.644193 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8" event={"ID":"f5652aad-c74e-4732-8bd7-1d814d64be92","Type":"ContainerDied","Data":"9b014d8a4b254591ac921c5efb64bec7d105a6e8799ebffe42ea0435e4dea529"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.648258 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.648324 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.651939 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" event={"ID":"52995638-b72e-460b-8306-f442c213ff19","Type":"ContainerDied","Data":"a51ca5669309ac508fbdd2c06129d1c0c12da144a05731785944f8a3eef40cff"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.652028 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.654890 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" event={"ID":"4b1744f9-9ed1-4e7b-a359-8a1499160126","Type":"ContainerDied","Data":"08b17320cec58d4aa6a4a8cf298871bdbeff031ff70594c09521eaa223e143ec"} Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.655019 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.658554 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.658620 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.661663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.658718 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.669951 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.669927 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.669966 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.669979 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.669989 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.669999 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670007 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670016 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv6bv\" (UniqueName: \"kubernetes.io/projected/56d233c6-d1fc-4b59-8998-afe61144f065-kube-api-access-cv6bv\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670028 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.670042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670040 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670052 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d233c6-d1fc-4b59-8998-afe61144f065-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670516 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670529 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6b7m\" (UniqueName: \"kubernetes.io/projected/52995638-b72e-460b-8306-f442c213ff19-kube-api-access-x6b7m\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670545 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d233c6-d1fc-4b59-8998-afe61144f065-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.670558 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52995638-b72e-460b-8306-f442c213ff19-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.685782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.685614 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.685782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.685743 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.695313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.695276 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.697845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.697793 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.705942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.705911 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.708529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.708466 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" start-of-body= Oct 30 20:10:27.708660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.708535 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="Get \"http://10.133.0.7:8181/ready\": dial tcp 10.133.0.7:8181: connect: connection refused" Oct 30 20:10:27.710204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.710161 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert" pod "52995638-b72e-460b-8306-f442c213ff19" (UID: "52995638-b72e-460b-8306-f442c213ff19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.725367 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.725231 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert" pod "56d233c6-d1fc-4b59-8998-afe61144f065" (UID: "56d233c6-d1fc-4b59-8998-afe61144f065"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.733393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.732852 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.736416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.736372 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.736416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.736399 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.747347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.747287 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.769426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.768727 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.769426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.769105 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9" pod "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" (UID: "35a6c5c6-c04a-40b6-a97a-828c0bd2d86c"). InnerVolumeSpecName "kube-api-access-v92f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.773186 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773156 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773186 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773187 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52995638-b72e-460b-8306-f442c213ff19-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773203 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773218 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92f9\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-kube-api-access-v92f9\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773234 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d233c6-d1fc-4b59-8998-afe61144f065-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773248 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773263 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773277 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773299 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773314 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52995638-b72e-460b-8306-f442c213ff19-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773328 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773345 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7489a135-99a1-41d0-8b09-1288c18ecf8b-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.773400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.773359 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.799313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.799256 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.802794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.802746 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.802956 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.802829 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.803176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.803141 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.803725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.803489 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.811003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.810967 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.812980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.812871 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.813631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.813610 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.818129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.817903 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.832836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.832779 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.832836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.832818 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.840710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.840647 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "kube-api-access-bbv8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.840831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.840751 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.845350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.845323 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 20:10:27.851195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.851165 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.856104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.856074 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-f6pvw"] Oct 30 20:10:27.865107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.865048 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.875350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875321 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875356 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875371 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90effab3-a893-40cd-b620-673575f1d679-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875386 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875400 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875415 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51892042-d2ae-4a69-9cc4-e347399249ed-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875430 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875443 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875458 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875472 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875486 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875499 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875511 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875526 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbv8n\" (UniqueName: \"kubernetes.io/projected/90effab3-a893-40cd-b620-673575f1d679-kube-api-access-bbv8n\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.875892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.875541 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:27.878241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.878213 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.879170 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.879131 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:10:27.887241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.879232 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.887241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.879271 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "kube-api-access-dmg7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.887241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.880501 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.888855 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.888823 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy" pod "7489a135-99a1-41d0-8b09-1288c18ecf8b" (UID: "7489a135-99a1-41d0-8b09-1288c18ecf8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.897195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.897163 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:10:27.900969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.900935 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.901229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.900980 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert" pod "90effab3-a893-40cd-b620-673575f1d679" (UID: "90effab3-a893-40cd-b620-673575f1d679"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:27.901229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.901093 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl" pod "51892042-d2ae-4a69-9cc4-e347399249ed" (UID: "51892042-d2ae-4a69-9cc4-e347399249ed"). InnerVolumeSpecName "kube-api-access-c7kgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:27.904259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.904214 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 20:10:27.910532 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.910511 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-qv2j2"] Oct 30 20:10:27.996552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.978493 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7kgl\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-kube-api-access-c7kgl\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.001547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:27.994633 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv"] Oct 30 20:10:28.002788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.001521 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.002922 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.002909 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51892042-d2ae-4a69-9cc4-e347399249ed-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003015 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003003 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90effab3-a893-40cd-b620-673575f1d679-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003115 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51892042-d2ae-4a69-9cc4-e347399249ed-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003205 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmg7z\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-kube-api-access-dmg7z\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003284 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51892042-d2ae-4a69-9cc4-e347399249ed-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003363 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90effab3-a893-40cd-b620-673575f1d679-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003439 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.003597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.003585 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7489a135-99a1-41d0-8b09-1288c18ecf8b-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:28.030919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.029738 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 20:10:28.046544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.046522 5733 scope.go:122] "RemoveContainer" containerID="30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd" Oct 30 20:10:28.068832 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.068798 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" path="/var/lib/kubelet/pods/3433ab80-4a84-4013-80d2-7c7555c470d3/volumes" Oct 30 20:10:28.070640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.070610 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" path="/var/lib/kubelet/pods/4aae276e-c30b-44e3-af8d-d88aa479cfb4/volumes" Oct 30 20:10:28.085147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.085107 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" path="/var/lib/kubelet/pods/4b1744f9-9ed1-4e7b-a359-8a1499160126/volumes" Oct 30 20:10:28.091675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.091650 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" path="/var/lib/kubelet/pods/cb994d3f-ac88-4fff-b9ee-400f5556cafb/volumes" Oct 30 20:10:28.092068 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.092039 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-tlh2h"] Oct 30 20:10:28.092136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.092094 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 20:10:28.092136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.092114 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-tx2n8"] Oct 30 20:10:28.115629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.115602 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 20:10:28.125176 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:28.123524 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b70387a_2ccb_4660_836b_9111b611df68.slice/crio-ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368 WatchSource:0}: Error finding container ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368: Status 404 returned error can't find the container with id ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368 Oct 30 20:10:28.128422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.128403 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3f57c5-openshift-default-6ff8995cc5-zdgnr"] Oct 30 20:10:28.190404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.187301 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 20:10:28.215728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.215655 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8a8207-openshift-default-6f5bb56fb7-sbzhg"] Oct 30 20:10:28.241244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.241218 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 20:10:28.250346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.250317 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-zj7hb"] Oct 30 20:10:28.258046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.258015 5733 scope.go:122] "RemoveContainer" containerID="e98008f879ac4beff479e8c78adfd03b818726f79887cd84a36feb340a6d2684" Oct 30 20:10:28.304812 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.304781 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 20:10:28.333001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.332008 5733 scope.go:122] "RemoveContainer" containerID="99161c544933d962fe7ffd9827eced0782c37807ef4eb3118699eb7dfb3af1a1" Oct 30 20:10:28.337992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.337952 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-2hbjs"] Oct 30 20:10:28.362153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.362122 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 20:10:28.377895 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.373436 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-gdnn8"] Oct 30 20:10:28.400820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.394131 5733 scope.go:122] "RemoveContainer" containerID="30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd" Oct 30 20:10:28.400820 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:28.394800 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd\": container with ID starting with 30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd not found: ID does not exist" containerID="30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd" Oct 30 20:10:28.400820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.394832 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd"} err="failed to get container status \"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd\": rpc error: code = NotFound desc = could not find container \"30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd\": container with ID starting with 30098a756e41d902d7764e77b5b46238a5b17b60ce97429b77eae28f58db64cd not found: ID does not exist" Oct 30 20:10:28.400820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.394855 5733 scope.go:122] "RemoveContainer" containerID="dab333a75bc732f29a57f9f02819c0a628d87bdec8733309ce2d719cd08572b8" Oct 30 20:10:28.441870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.441844 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 20:10:28.465990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.465924 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-kq7xq"] Oct 30 20:10:28.557204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.557181 5733 scope.go:122] "RemoveContainer" containerID="107db65fd094c0b5f82e71abc806daeab1926735e830876b97614a7a65397855" Oct 30 20:10:28.608489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.608446 5733 scope.go:122] "RemoveContainer" containerID="04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f" Oct 30 20:10:28.608868 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:28.608843 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f\": container with ID starting with 04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f not found: ID does not exist" containerID="04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f" Oct 30 20:10:28.609005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.608881 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f"} err="failed to get container status \"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f\": rpc error: code = NotFound desc = could not find container \"04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f\": container with ID starting with 04a16177cc4fc13922430259c5d89fe2fd7d8cf6b7260538c85388a333f01e3f not found: ID does not exist" Oct 30 20:10:28.609005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.608905 5733 scope.go:122] "RemoveContainer" containerID="6f5d15c9cde2e1e9e132f5ef785511cd6041e47025ae39e2b4f7ea2b4a9790bc" Oct 30 20:10:28.631216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.631179 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.168:8443/health\": dial tcp 10.133.0.168:8443: connect: connection refused" Oct 30 20:10:28.662156 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.662014 5733 scope.go:122] "RemoveContainer" containerID="971c24d0caa4087ea0b18f3e1a5fed49f146c79ec5060061df0705ba51390b26" Oct 30 20:10:28.665746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.665704 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:10:28.674201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.674172 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" event={"ID":"1b70387a-2ccb-4660-836b-9111b611df68","Type":"ContainerStarted","Data":"ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368"} Oct 30 20:10:28.676448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.676422 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerStarted","Data":"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea"} Oct 30 20:10:28.676555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.676501 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:10:28.680050 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.680026 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerStarted","Data":"47bfd6642f014f09aebab79a5592d5ed573a3601a186431685262be4093049e0"} Oct 30 20:10:28.680188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.680169 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:28.688722 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.688704 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerStarted","Data":"f49301dd7eca1a9597e87b1bbfbfef3375a3bcf03f43d03b612f196352da746f"} Oct 30 20:10:28.688825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.688743 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:28.692164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.692140 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerStarted","Data":"0374c4c11664ec8670fd204cdb316d169f4f1ec81fc7ba1ac3af039ebcb5feac"} Oct 30 20:10:28.692284 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.692212 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:10:28.697172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.697154 5733 scope.go:122] "RemoveContainer" containerID="f1f15dca5e73b7c55271a553ff58fdc66d89066c623b6b4086ada73b2da21a60" Oct 30 20:10:28.698135 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.698109 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerStarted","Data":"5cb3f69e00b4e951db23cb6ee00f063211125d1ecba3bab270496754f56b9d69"} Oct 30 20:10:28.698305 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.698286 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:28.743000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.742976 5733 scope.go:122] "RemoveContainer" containerID="4c4761934862cdc7a3f6b2cc0d1c5b9799955b1bb161babf36a63c9a37329ce6" Oct 30 20:10:28.810474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.810447 5733 scope.go:122] "RemoveContainer" containerID="df3346d191c5ea28010f0a63c829ddc9995698d9f92f0081cf32b9f2acd89225" Oct 30 20:10:28.848950 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:28.848910 5733 scope.go:122] "RemoveContainer" containerID="432546abde812b81a38c559ac44aedba642da56220811edccaeeb6b39c49d08d" Oct 30 20:10:29.221341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:29.221298 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:29.381233 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:29.381199 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:10:29.723539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:29.723458 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" event={"ID":"1b70387a-2ccb-4660-836b-9111b611df68","Type":"ContainerStarted","Data":"a86f772ed299cd967a2b343596a6469607867289cf54ca127e247f90f1f0425d"} Oct 30 20:10:29.725791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:29.725761 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" event={"ID":"82727e18-39c9-438c-b335-259eb7281109","Type":"ContainerStarted","Data":"f8fcc068fc77b6b409d318fea26b24eedf761db86d971dea3026da08bb13eb58"} Oct 30 20:10:29.725965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:29.725947 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 20:10:29.765425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:29.765377 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" podStartSLOduration=8.765363428 podStartE2EDuration="8.765363428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:29.759567596 +0000 UTC m=+2389.827403165" watchObservedRunningTime="2026-10-30 20:10:29.765363428 +0000 UTC m=+2389.833198995" Oct 30 20:10:30.066669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.066631 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" path="/var/lib/kubelet/pods/35a6c5c6-c04a-40b6-a97a-828c0bd2d86c/volumes" Oct 30 20:10:30.067349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.067332 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" path="/var/lib/kubelet/pods/51892042-d2ae-4a69-9cc4-e347399249ed/volumes" Oct 30 20:10:30.067832 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.067818 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52995638-b72e-460b-8306-f442c213ff19" path="/var/lib/kubelet/pods/52995638-b72e-460b-8306-f442c213ff19/volumes" Oct 30 20:10:30.124583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.124548 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" path="/var/lib/kubelet/pods/56d233c6-d1fc-4b59-8998-afe61144f065/volumes" Oct 30 20:10:30.174684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.174649 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" path="/var/lib/kubelet/pods/7489a135-99a1-41d0-8b09-1288c18ecf8b/volumes" Oct 30 20:10:30.220887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.220764 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90effab3-a893-40cd-b620-673575f1d679" path="/var/lib/kubelet/pods/90effab3-a893-40cd-b620-673575f1d679/volumes" Oct 30 20:10:30.221514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.221496 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" path="/var/lib/kubelet/pods/b9558d5f-e925-43dc-90b4-928cd373c624/volumes" Oct 30 20:10:30.227702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.227681 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" path="/var/lib/kubelet/pods/f5652aad-c74e-4732-8bd7-1d814d64be92/volumes" Oct 30 20:10:30.708967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.708933 5733 patch_prober.go:29] interesting pod/dns-default-bjt85 container/dns namespace/openshift-dns: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=kubernetes Oct 30 20:10:30.712854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.708991 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-bjt85" podUID="b107b211-0176-4170-8d0f-85d4b36a4f30" containerName="dns" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:10:30.746608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.746581 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:10:30.746776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.746708 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"14bc109f3f37390f9caec5249c61d474f48040fd9d7e0778a114a37f79697119"} Oct 30 20:10:30.746843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.746773 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ng7cn" event={"ID":"d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5","Type":"ContainerStarted","Data":"645f311109445e47c93b934f5466223e9996288443c7c3f1f4b20ad2b1c9196c"} Oct 30 20:10:30.748608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.748578 5733 generic.go:356] "Generic (PLEG): container finished" podID="1b70387a-2ccb-4660-836b-9111b611df68" containerID="a86f772ed299cd967a2b343596a6469607867289cf54ca127e247f90f1f0425d" exitCode=0 Oct 30 20:10:30.748768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.748613 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" event={"ID":"1b70387a-2ccb-4660-836b-9111b611df68","Type":"ContainerDied","Data":"a86f772ed299cd967a2b343596a6469607867289cf54ca127e247f90f1f0425d"} Oct 30 20:10:30.751036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.751013 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bjt85" event={"ID":"b107b211-0176-4170-8d0f-85d4b36a4f30","Type":"ContainerStarted","Data":"913683497da206cdb9ec23ee08ba1e3dd33cd7c530f9de47beb8df87d7c0f68c"} Oct 30 20:10:30.751195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.751070 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:30.856853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:30.856652 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.561946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.561864 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" containerID="cri-o://3714c0014297877970558d020c838c846f37d5b13e99c3bf65564e5837da6db9" gracePeriod=25 Oct 30 20:10:31.785391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.785357 5733 generic.go:356] "Generic (PLEG): container finished" podID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerID="3714c0014297877970558d020c838c846f37d5b13e99c3bf65564e5837da6db9" exitCode=0 Oct 30 20:10:31.785551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.785434 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" event={"ID":"8ed984b5-27dc-47f6-b3a3-0be88a09763e","Type":"ContainerDied","Data":"3714c0014297877970558d020c838c846f37d5b13e99c3bf65564e5837da6db9"} Oct 30 20:10:31.787713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.787678 5733 generic.go:356] "Generic (PLEG): container finished" podID="63919396-52a0-4723-9d83-dd3849acdbc1" containerID="174137be6cb2f13be5c30546e48c66ee0b0c6139b4848b7fe4d1076cebb55397" exitCode=0 Oct 30 20:10:31.787871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.787760 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerDied","Data":"174137be6cb2f13be5c30546e48c66ee0b0c6139b4848b7fe4d1076cebb55397"} Oct 30 20:10:31.788700 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.788499 5733 scope.go:122] "RemoveContainer" containerID="174137be6cb2f13be5c30546e48c66ee0b0c6139b4848b7fe4d1076cebb55397" Oct 30 20:10:31.791070 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.791025 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:10:31.791179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.791148 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:10:31.791254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.791204 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:10:31.908290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.907174 5733 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:10:31.931368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.930970 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:10:31.986120 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.986087 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\" (UniqueName: \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\") pod \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " Oct 30 20:10:31.986604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.986572 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\") pod \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\" (UID: \"8ed984b5-27dc-47f6-b3a3-0be88a09763e\") " Oct 30 20:10:31.989691 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:31.989661 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume" pod "8ed984b5-27dc-47f6-b3a3-0be88a09763e" (UID: "8ed984b5-27dc-47f6-b3a3-0be88a09763e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:32.002217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.002179 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t" pod "8ed984b5-27dc-47f6-b3a3-0be88a09763e" (UID: "8ed984b5-27dc-47f6-b3a3-0be88a09763e"). InnerVolumeSpecName "kube-api-access-xtj4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:32.018946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.018924 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:32.095654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.091992 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\" (UniqueName: \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\") pod \"1b70387a-2ccb-4660-836b-9111b611df68\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " Oct 30 20:10:32.095902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.095875 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\") pod \"1b70387a-2ccb-4660-836b-9111b611df68\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " Oct 30 20:10:32.096026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.095939 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\") pod \"1b70387a-2ccb-4660-836b-9111b611df68\" (UID: \"1b70387a-2ccb-4660-836b-9111b611df68\") " Oct 30 20:10:32.106731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.104484 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca" pod "1b70387a-2ccb-4660-836b-9111b611df68" (UID: "1b70387a-2ccb-4660-836b-9111b611df68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:32.106731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.105082 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:32.106731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.105113 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ed984b5-27dc-47f6-b3a3-0be88a09763e-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:32.106731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.105128 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtj4t\" (UniqueName: \"kubernetes.io/projected/8ed984b5-27dc-47f6-b3a3-0be88a09763e-kube-api-access-xtj4t\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:32.116757 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.116475 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca" pod "1b70387a-2ccb-4660-836b-9111b611df68" (UID: "1b70387a-2ccb-4660-836b-9111b611df68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:32.121588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.117608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:32.121588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.121195 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp" pod "1b70387a-2ccb-4660-836b-9111b611df68" (UID: "1b70387a-2ccb-4660-836b-9111b611df68"). InnerVolumeSpecName "kube-api-access-9klwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:32.207233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.207201 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9klwp\" (UniqueName: \"kubernetes.io/projected/1b70387a-2ccb-4660-836b-9111b611df68-kube-api-access-9klwp\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:32.207233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.207228 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b70387a-2ccb-4660-836b-9111b611df68-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:32.657953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.657869 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-glbct" Oct 30 20:10:32.807588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.807559 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" Oct 30 20:10:32.807802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.807591 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc" event={"ID":"8ed984b5-27dc-47f6-b3a3-0be88a09763e","Type":"ContainerDied","Data":"c6bd2fc509ab2dd73a0784824be54f1def5924c2a1b9192cc39704525e85f96c"} Oct 30 20:10:32.807802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.807684 5733 scope.go:122] "RemoveContainer" containerID="3714c0014297877970558d020c838c846f37d5b13e99c3bf65564e5837da6db9" Oct 30 20:10:32.809535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.809513 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" Oct 30 20:10:32.809674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.809509 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv" event={"ID":"1b70387a-2ccb-4660-836b-9111b611df68","Type":"ContainerDied","Data":"ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368"} Oct 30 20:10:32.809674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.809623 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced69ae3c04947c81afed29b498521c2f4c3131933a3fef0a14c6547fa347368" Oct 30 20:10:32.812437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.812415 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerStarted","Data":"5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa"} Oct 30 20:10:32.826476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.826406 5733 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:10:31.907191518Z","UUID":"3c9f5360-7894-402c-a429-160fdef1fa58","Handler":null,"Name":"","Endpoint":""} Oct 30 20:10:32.827472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.827449 5733 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:10:32.827625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.827490 5733 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:10:32.885613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.885583 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:10:32.893640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:32.893610 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7f7d21-5cbd9cd4f4-8r6mc"] Oct 30 20:10:33.292570 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:33.292538 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.495197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:33.495162 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:10:33.789128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:33.789090 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:10:33.794906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:33.794881 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:10:33.824081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:33.824043 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:10:34.053141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.053046 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" path="/var/lib/kubelet/pods/8ed984b5-27dc-47f6-b3a3-0be88a09763e/volumes" Oct 30 20:10:34.601500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.601450 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:34.731602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.731572 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:10:34.731979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.731940 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:10:34.732196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.732175 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:10:34.732498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.732481 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:10:34.732980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:34.732958 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:10:35.750739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:35.750707 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:35.980116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:35.980008 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.230124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230082 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl"] Oct 30 20:10:36.230906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230888 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" Oct 30 20:10:36.230966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230907 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" Oct 30 20:10:36.230966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230921 5733 cpu_manager.go:414] "containerMap: removing container" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:36.230966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230927 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:36.230966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230943 5733 cpu_manager.go:414] "containerMap: removing container" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" Oct 30 20:10:36.230966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230948 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230969 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230974 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230980 5733 cpu_manager.go:414] "containerMap: removing container" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230985 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230993 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.230997 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231003 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231008 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231017 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231022 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231033 5733 cpu_manager.go:414] "containerMap: removing container" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231037 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231047 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231052 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231077 5733 cpu_manager.go:414] "containerMap: removing container" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231083 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231094 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231098 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231109 5733 cpu_manager.go:414] "containerMap: removing container" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231113 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231126 5733 cpu_manager.go:414] "containerMap: removing container" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231131 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231143 5733 cpu_manager.go:414] "containerMap: removing container" podUID="1b70387a-2ccb-4660-836b-9111b611df68" containerName="revoke-keys" Oct 30 20:10:36.231167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231148 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b70387a-2ccb-4660-836b-9111b611df68" containerName="revoke-keys" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231258 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1744f9-9ed1-4e7b-a359-8a1499160126" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231268 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5652aad-c74e-4732-8bd7-1d814d64be92" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231275 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9558d5f-e925-43dc-90b4-928cd373c624" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231282 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="51892042-d2ae-4a69-9cc4-e347399249ed" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231289 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed984b5-27dc-47f6-b3a3-0be88a09763e" containerName="payload-processing" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231295 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231301 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7489a135-99a1-41d0-8b09-1288c18ecf8b" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231307 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d233c6-d1fc-4b59-8998-afe61144f065" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231315 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="3433ab80-4a84-4013-80d2-7c7555c470d3" containerName="payload-processing" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231321 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231327 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b70387a-2ccb-4660-836b-9111b611df68" containerName="revoke-keys" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231333 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aae276e-c30b-44e3-af8d-d88aa479cfb4" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231338 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231344 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="35a6c5c6-c04a-40b6-a97a-828c0bd2d86c" containerName="istio-proxy" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231353 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb994d3f-ac88-4fff-b9ee-400f5556cafb" containerName="payload-pre-processing" Oct 30 20:10:36.231952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.231359 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="52995638-b72e-460b-8306-f442c213ff19" containerName="istio-proxy" Oct 30 20:10:36.235297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.235273 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.243349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.243330 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:36.243449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.243393 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:10:36.322889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.322777 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl"] Oct 30 20:10:36.388799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.388761 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.389007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.388935 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5857\" (UniqueName: \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.389240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.389219 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.491262 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.491184 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.491406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.491277 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.491406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.491356 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5857\" (UniqueName: \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.491840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.491819 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.493758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.493730 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.510133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.510104 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5857\" (UniqueName: \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\") pod \"maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.549653 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.549626 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:36.825203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.822337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.876247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:36.876219 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl"] Oct 30 20:10:36.877097 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:36.877032 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c78ece5_ccf1_43ea_9b24_038af64577da.slice/crio-001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db WatchSource:0}: Error finding container 001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db: Status 404 returned error can't find the container with id 001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db Oct 30 20:10:37.064886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.064851 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4"] Oct 30 20:10:37.065694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.065677 5733 cpu_manager.go:414] "containerMap: removing container" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:37.065694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.065694 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90effab3-a893-40cd-b620-673575f1d679" containerName="istio-proxy" Oct 30 20:10:37.070756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.070738 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.088811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.088786 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4"] Oct 30 20:10:37.109987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.109957 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.110327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.110249 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.110327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.110308 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.211414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.211386 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.211596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.211442 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.211596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.211513 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.212202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.212179 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.213878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.213848 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.238213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.238188 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\") pod \"maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.386480 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.386447 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:37.626112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.626046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:37.659192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.659116 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:10:37.659346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.659195 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-2s5r5" Oct 30 20:10:37.662157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.662135 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-64d7d576f9-fb9qh" Oct 30 20:10:37.675892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.675868 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4"] Oct 30 20:10:37.676867 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:37.676838 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0512083_698e_4061_b1c9_689faaf7dda7.slice/crio-e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b WatchSource:0}: Error finding container e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b: Status 404 returned error can't find the container with id e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b Oct 30 20:10:37.881007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.880973 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" event={"ID":"5c78ece5-ccf1-43ea-9b24-038af64577da","Type":"ContainerStarted","Data":"996aec9e2c094afa81459ff3e81bf6e67b8c9ed1514ba598e728654841df7715"} Oct 30 20:10:37.881519 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.881039 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" event={"ID":"5c78ece5-ccf1-43ea-9b24-038af64577da","Type":"ContainerStarted","Data":"001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db"} Oct 30 20:10:37.882608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.882577 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" event={"ID":"c0512083-698e-4061-b1c9-689faaf7dda7","Type":"ContainerStarted","Data":"e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b"} Oct 30 20:10:37.929689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:37.929639 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" podStartSLOduration=1.929624482 podStartE2EDuration="1.929624482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:37.914706818 +0000 UTC m=+2397.982542386" watchObservedRunningTime="2026-10-30 20:10:37.929624482 +0000 UTC m=+2397.997460054" Oct 30 20:10:38.894078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:38.893971 5733 generic.go:356] "Generic (PLEG): container finished" podID="c0512083-698e-4061-b1c9-689faaf7dda7" containerID="d022ea1dd3925036e56c9cfa705c0f27539e170ecf434665b030bb36553e6564" exitCode=0 Oct 30 20:10:38.894078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:38.894025 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" event={"ID":"c0512083-698e-4061-b1c9-689faaf7dda7","Type":"ContainerDied","Data":"d022ea1dd3925036e56c9cfa705c0f27539e170ecf434665b030bb36553e6564"} Oct 30 20:10:38.896122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:38.896097 5733 generic.go:356] "Generic (PLEG): container finished" podID="5c78ece5-ccf1-43ea-9b24-038af64577da" containerID="996aec9e2c094afa81459ff3e81bf6e67b8c9ed1514ba598e728654841df7715" exitCode=0 Oct 30 20:10:38.896273 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:38.896133 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" event={"ID":"5c78ece5-ccf1-43ea-9b24-038af64577da","Type":"ContainerDied","Data":"996aec9e2c094afa81459ff3e81bf6e67b8c9ed1514ba598e728654841df7715"} Oct 30 20:10:39.104416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:39.104385 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:40.152695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.152659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:40.156397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.156319 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:40.168361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.168327 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:40.265735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.265689 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\") pod \"5c78ece5-ccf1-43ea-9b24-038af64577da\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " Oct 30 20:10:40.265925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.265860 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\") pod \"5c78ece5-ccf1-43ea-9b24-038af64577da\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " Oct 30 20:10:40.265925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.265912 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\" (UniqueName: \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\") pod \"5c78ece5-ccf1-43ea-9b24-038af64577da\" (UID: \"5c78ece5-ccf1-43ea-9b24-038af64577da\") " Oct 30 20:10:40.266131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.265962 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\") pod \"c0512083-698e-4061-b1c9-689faaf7dda7\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " Oct 30 20:10:40.266131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.265999 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\") pod \"c0512083-698e-4061-b1c9-689faaf7dda7\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " Oct 30 20:10:40.266131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.266092 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\") pod \"c0512083-698e-4061-b1c9-689faaf7dda7\" (UID: \"c0512083-698e-4061-b1c9-689faaf7dda7\") " Oct 30 20:10:40.266349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.266311 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca" pod "5c78ece5-ccf1-43ea-9b24-038af64577da" (UID: "5c78ece5-ccf1-43ea-9b24-038af64577da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:40.266413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.266357 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca" pod "c0512083-698e-4061-b1c9-689faaf7dda7" (UID: "c0512083-698e-4061-b1c9-689faaf7dda7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:40.266993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.266964 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.267149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.266997 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.268227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.268201 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca" pod "5c78ece5-ccf1-43ea-9b24-038af64577da" (UID: "5c78ece5-ccf1-43ea-9b24-038af64577da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:40.268336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.268247 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca" pod "c0512083-698e-4061-b1c9-689faaf7dda7" (UID: "c0512083-698e-4061-b1c9-689faaf7dda7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:40.268336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.268290 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857" pod "5c78ece5-ccf1-43ea-9b24-038af64577da" (UID: "5c78ece5-ccf1-43ea-9b24-038af64577da"). InnerVolumeSpecName "kube-api-access-s5857". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:40.268336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.268305 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm" pod "c0512083-698e-4061-b1c9-689faaf7dda7" (UID: "c0512083-698e-4061-b1c9-689faaf7dda7"). InnerVolumeSpecName "kube-api-access-d6wpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:40.368272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.368240 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5857\" (UniqueName: \"kubernetes.io/projected/5c78ece5-ccf1-43ea-9b24-038af64577da-kube-api-access-s5857\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.368272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.368269 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0512083-698e-4061-b1c9-689faaf7dda7-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.368272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.368280 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6wpm\" (UniqueName: \"kubernetes.io/projected/c0512083-698e-4061-b1c9-689faaf7dda7-kube-api-access-d6wpm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.368496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.368291 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c78ece5-ccf1-43ea-9b24-038af64577da-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:40.753174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.753145 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85768f955d-hxzvg" Oct 30 20:10:40.790407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.790382 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bjt85" Oct 30 20:10:40.816103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.816074 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:10:40.816244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.816075 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:10:40.920227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.920195 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" Oct 30 20:10:40.928397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.928366 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4" event={"ID":"c0512083-698e-4061-b1c9-689faaf7dda7","Type":"ContainerDied","Data":"e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b"} Oct 30 20:10:40.928604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.928455 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63efad511160e66a59d61bd399c3a09d1692f4b8cffef60b38d4f981826535b" Oct 30 20:10:40.929948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.929687 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" event={"ID":"5c78ece5-ccf1-43ea-9b24-038af64577da","Type":"ContainerDied","Data":"001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db"} Oct 30 20:10:40.929948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.929732 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="001c748b256fec827f3d7f7796b028934ecdcabfa5e3501c144d0f2a4801b1db" Oct 30 20:10:40.929948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:40.929811 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl" Oct 30 20:10:41.238082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:41.233925 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:41.389657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:41.389621 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:10:41.549199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:41.549102 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:41.881075 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:41.878677 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:42.789374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:42.789341 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:43.871079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:43.870559 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:44.889548 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:44.889453 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.141499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:45.141404 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.304726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:45.304691 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.458286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:45.458197 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:46.139931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.139892 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:46.279297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.279266 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:10:46.292412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.292381 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:10:46.390760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.390671 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" containerID="cri-o://492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502" gracePeriod=25 Oct 30 20:10:46.419569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.419547 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:46.617279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.617138 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:46.954874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:46.954837 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:10:47.000338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.000292 5733 generic.go:356] "Generic (PLEG): container finished" podID="058c7110-65ab-4311-9688-b389e30c5f2a" containerID="492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502" exitCode=0 Oct 30 20:10:47.000606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.000578 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" event={"ID":"058c7110-65ab-4311-9688-b389e30c5f2a","Type":"ContainerDied","Data":"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502"} Oct 30 20:10:47.001021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.000991 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" event={"ID":"058c7110-65ab-4311-9688-b389e30c5f2a","Type":"ContainerDied","Data":"51a10eaddd3cbf5a769b7e2dbe1f2f1dda892ba1fa9eaa1b5bdcf04ccaaa1287"} Oct 30 20:10:47.001172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.001039 5733 scope.go:122] "RemoveContainer" containerID="492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502" Oct 30 20:10:47.001172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.000924 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88" Oct 30 20:10:47.027630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.027605 5733 scope.go:122] "RemoveContainer" containerID="492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502" Oct 30 20:10:47.028275 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:47.028239 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502\": container with ID starting with 492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502 not found: ID does not exist" containerID="492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502" Oct 30 20:10:47.028380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.028285 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502"} err="failed to get container status \"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502\": rpc error: code = NotFound desc = could not find container \"492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502\": container with ID starting with 492ef7213878e53fd8b65a8dcb0934f5a9f333706e29e479ac6b70b9e63a0502 not found: ID does not exist" Oct 30 20:10:47.086104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.085725 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\" (UniqueName: \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\") pod \"058c7110-65ab-4311-9688-b389e30c5f2a\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " Oct 30 20:10:47.086104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.085843 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\") pod \"058c7110-65ab-4311-9688-b389e30c5f2a\" (UID: \"058c7110-65ab-4311-9688-b389e30c5f2a\") " Oct 30 20:10:47.088089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.087732 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume" pod "058c7110-65ab-4311-9688-b389e30c5f2a" (UID: "058c7110-65ab-4311-9688-b389e30c5f2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:47.110095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.110029 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx" pod "058c7110-65ab-4311-9688-b389e30c5f2a" (UID: "058c7110-65ab-4311-9688-b389e30c5f2a"). InnerVolumeSpecName "kube-api-access-fcbpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:47.188715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.188678 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcbpx\" (UniqueName: \"kubernetes.io/projected/058c7110-65ab-4311-9688-b389e30c5f2a-kube-api-access-fcbpx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:47.188715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.188714 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058c7110-65ab-4311-9688-b389e30c5f2a-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:47.208732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.208689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:47.242601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.242521 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:10:47.297325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.297284 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:10:47.387839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.387155 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:10:47.415117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.397235 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f5eee-d4f4c6bc-nxp88"] Oct 30 20:10:47.563471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.563256 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:47.793136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:47.775342 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:48.067119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:48.066421 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" path="/var/lib/kubelet/pods/058c7110-65ab-4311-9688-b389e30c5f2a/volumes" Oct 30 20:10:48.788167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:48.788126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:49.959101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:49.959020 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:51.061936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.061894 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:51.280108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.280046 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" containerID="cri-o://ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c" gracePeriod=25 Oct 30 20:10:51.292789 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.292726 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" containerID="cri-o://e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32" gracePeriod=25 Oct 30 20:10:51.636021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.635993 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:10:51.674698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.674666 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:10:51.687927 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.687890 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\") pod \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " Oct 30 20:10:51.688089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.687941 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\" (UniqueName: \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\") pod \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " Oct 30 20:10:51.688089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.688008 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\" (UniqueName: \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\") pod \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\" (UID: \"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe\") " Oct 30 20:10:51.688232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.688099 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\") pod \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\" (UID: \"8b964b0d-e172-433f-b5cb-7fd19823d2d7\") " Oct 30 20:10:51.688424 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.688396 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume" pod "f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" (UID: "f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:51.688717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.688695 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume" pod "8b964b0d-e172-433f-b5cb-7fd19823d2d7" (UID: "8b964b0d-e172-433f-b5cb-7fd19823d2d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:51.689122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.689102 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:51.689219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.689125 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b964b0d-e172-433f-b5cb-7fd19823d2d7-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:51.690999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.690979 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds" pod "8b964b0d-e172-433f-b5cb-7fd19823d2d7" (UID: "8b964b0d-e172-433f-b5cb-7fd19823d2d7"). InnerVolumeSpecName "kube-api-access-5thds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:51.691123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.691022 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg" pod "f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" (UID: "f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe"). InnerVolumeSpecName "kube-api-access-l8xwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:51.790352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.790317 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thds\" (UniqueName: \"kubernetes.io/projected/8b964b0d-e172-433f-b5cb-7fd19823d2d7-kube-api-access-5thds\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:51.790352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:51.790345 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8xwg\" (UniqueName: \"kubernetes.io/projected/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe-kube-api-access-l8xwg\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:52.012727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.011126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:52.083914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.083870 5733 generic.go:356] "Generic (PLEG): container finished" podID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerID="e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32" exitCode=0 Oct 30 20:10:52.084434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.083930 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" Oct 30 20:10:52.084434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.083930 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" event={"ID":"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe","Type":"ContainerDied","Data":"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32"} Oct 30 20:10:52.084434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.084001 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7" event={"ID":"f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe","Type":"ContainerDied","Data":"4d67256facb16c6108269fb664dc23fcf0aea64f7320224c448232e3f310c33f"} Oct 30 20:10:52.084434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.084034 5733 scope.go:122] "RemoveContainer" containerID="e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32" Oct 30 20:10:52.086867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.086828 5733 generic.go:356] "Generic (PLEG): container finished" podID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerID="ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c" exitCode=0 Oct 30 20:10:52.087267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.087246 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" event={"ID":"8b964b0d-e172-433f-b5cb-7fd19823d2d7","Type":"ContainerDied","Data":"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c"} Oct 30 20:10:52.087407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.087393 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" event={"ID":"8b964b0d-e172-433f-b5cb-7fd19823d2d7","Type":"ContainerDied","Data":"7922264d1737215f871ba23437aeed8f2343405453f4e4f12ac9a199d7104f75"} Oct 30 20:10:52.087572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.087559 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx" Oct 30 20:10:52.108305 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.105244 5733 scope.go:122] "RemoveContainer" containerID="e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32" Oct 30 20:10:52.110865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.110052 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:10:52.112026 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:52.111991 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32\": container with ID starting with e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32 not found: ID does not exist" containerID="e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32" Oct 30 20:10:52.112185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.112075 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32"} err="failed to get container status \"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32\": rpc error: code = NotFound desc = could not find container \"e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32\": container with ID starting with e4590db1a4bbbcb15d9bc9b008bf9fc0ada909ec5eecfaa7b84d9a15aea6fe32 not found: ID does not exist" Oct 30 20:10:52.112185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.112106 5733 scope.go:122] "RemoveContainer" containerID="ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c" Oct 30 20:10:52.116150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.116126 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-19f2ee-64467767dc-zflq7"] Oct 30 20:10:52.124051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.124029 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:10:52.130144 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.130119 5733 scope.go:122] "RemoveContainer" containerID="ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c" Oct 30 20:10:52.130543 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:52.130509 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c\": container with ID starting with ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c not found: ID does not exist" containerID="ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c" Oct 30 20:10:52.130636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.130549 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c"} err="failed to get container status \"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c\": rpc error: code = NotFound desc = could not find container \"ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c\": container with ID starting with ae639c6fc733e25029615b68c56d898d9c6254b9d188714800e77a374256666c not found: ID does not exist" Oct 30 20:10:52.131675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.131655 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-19f2ee-868c6d594b-9vnrx"] Oct 30 20:10:52.243100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.243022 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" containerID="cri-o://db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461" gracePeriod=25 Oct 30 20:10:52.298328 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.298259 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" containerID="cri-o://eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f" gracePeriod=25 Oct 30 20:10:52.541357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.541305 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:52.621274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.621244 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:10:52.698112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.698038 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:10:52.810497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810466 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\") pod \"dc31087c-8882-44ff-9240-42cf43576885\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " Oct 30 20:10:52.810701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810527 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\" (UniqueName: \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\") pod \"dc31087c-8882-44ff-9240-42cf43576885\" (UID: \"dc31087c-8882-44ff-9240-42cf43576885\") " Oct 30 20:10:52.810701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810561 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\") pod \"4b80ef9e-12fc-48a1-a912-891358436d57\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " Oct 30 20:10:52.810846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810711 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\" (UniqueName: \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\") pod \"4b80ef9e-12fc-48a1-a912-891358436d57\" (UID: \"4b80ef9e-12fc-48a1-a912-891358436d57\") " Oct 30 20:10:52.811003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810972 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume" pod "4b80ef9e-12fc-48a1-a912-891358436d57" (UID: "4b80ef9e-12fc-48a1-a912-891358436d57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:52.811003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.810989 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume" pod "dc31087c-8882-44ff-9240-42cf43576885" (UID: "dc31087c-8882-44ff-9240-42cf43576885"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:52.811474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.811450 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b80ef9e-12fc-48a1-a912-891358436d57-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:52.811474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.811473 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc31087c-8882-44ff-9240-42cf43576885-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:52.812879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.812854 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf" pod "dc31087c-8882-44ff-9240-42cf43576885" (UID: "dc31087c-8882-44ff-9240-42cf43576885"). InnerVolumeSpecName "kube-api-access-5l9wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:52.812964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.812924 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr" pod "4b80ef9e-12fc-48a1-a912-891358436d57" (UID: "4b80ef9e-12fc-48a1-a912-891358436d57"). InnerVolumeSpecName "kube-api-access-mgbzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:52.913438 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.913402 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgbzr\" (UniqueName: \"kubernetes.io/projected/4b80ef9e-12fc-48a1-a912-891358436d57-kube-api-access-mgbzr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:52.913593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.913448 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l9wf\" (UniqueName: \"kubernetes.io/projected/dc31087c-8882-44ff-9240-42cf43576885-kube-api-access-5l9wf\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:52.979686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:52.979638 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.098269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.098236 5733 generic.go:356] "Generic (PLEG): container finished" podID="4b80ef9e-12fc-48a1-a912-891358436d57" containerID="eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f" exitCode=0 Oct 30 20:10:53.098739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.098288 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" Oct 30 20:10:53.098739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.098322 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" event={"ID":"4b80ef9e-12fc-48a1-a912-891358436d57","Type":"ContainerDied","Data":"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f"} Oct 30 20:10:53.098739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.098389 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq" event={"ID":"4b80ef9e-12fc-48a1-a912-891358436d57","Type":"ContainerDied","Data":"22b5deb72a3f126d7bf52808efb65ce28498b07c9b0d02721a1bcac1d35e2fa5"} Oct 30 20:10:53.098739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.098410 5733 scope.go:122] "RemoveContainer" containerID="eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f" Oct 30 20:10:53.101067 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.101027 5733 generic.go:356] "Generic (PLEG): container finished" podID="dc31087c-8882-44ff-9240-42cf43576885" containerID="db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461" exitCode=0 Oct 30 20:10:53.101210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.101088 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" Oct 30 20:10:53.101210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.101085 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" event={"ID":"dc31087c-8882-44ff-9240-42cf43576885","Type":"ContainerDied","Data":"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461"} Oct 30 20:10:53.101340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.101216 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm" event={"ID":"dc31087c-8882-44ff-9240-42cf43576885","Type":"ContainerDied","Data":"ed94facc5056f06a1da49c606c079851df28c8d03d86d10e799a309970a0a479"} Oct 30 20:10:53.111845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.111823 5733 scope.go:122] "RemoveContainer" containerID="eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f" Oct 30 20:10:53.112266 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:53.112245 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f\": container with ID starting with eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f not found: ID does not exist" containerID="eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f" Oct 30 20:10:53.112364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.112272 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f"} err="failed to get container status \"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f\": rpc error: code = NotFound desc = could not find container \"eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f\": container with ID starting with eea55a67e7df1ee4081d55a7096f8b9a24282af3ef19ba5411ba1289e014986f not found: ID does not exist" Oct 30 20:10:53.112364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.112289 5733 scope.go:122] "RemoveContainer" containerID="db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461" Oct 30 20:10:53.124148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.124130 5733 scope.go:122] "RemoveContainer" containerID="db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461" Oct 30 20:10:53.124463 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:53.124438 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461\": container with ID starting with db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461 not found: ID does not exist" containerID="db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461" Oct 30 20:10:53.124530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.124475 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461"} err="failed to get container status \"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461\": rpc error: code = NotFound desc = could not find container \"db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461\": container with ID starting with db687127d8e4c293fffbbfd2f4c508307e990dd69d76ec288ee644f6bce73461 not found: ID does not exist" Oct 30 20:10:53.132259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.132237 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:10:53.139593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.139571 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1f246b-6757fd8c9f-g8nkq"] Oct 30 20:10:53.147969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.147767 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:10:53.154660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.154627 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1f246b-bcf449946-lbpnm"] Oct 30 20:10:53.303554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.303518 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.485764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.485666 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.689763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.689725 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.840142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:53.839271 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:54.055262 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.055226 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" path="/var/lib/kubelet/pods/4b80ef9e-12fc-48a1-a912-891358436d57/volumes" Oct 30 20:10:54.055785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.055767 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" path="/var/lib/kubelet/pods/8b964b0d-e172-433f-b5cb-7fd19823d2d7/volumes" Oct 30 20:10:54.056275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.056259 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc31087c-8882-44ff-9240-42cf43576885" path="/var/lib/kubelet/pods/dc31087c-8882-44ff-9240-42cf43576885/volumes" Oct 30 20:10:54.056705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.056690 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" path="/var/lib/kubelet/pods/f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe/volumes" Oct 30 20:10:54.174743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.174699 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p"] Oct 30 20:10:54.175576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175549 5733 cpu_manager.go:414] "containerMap: removing container" podUID="c0512083-698e-4061-b1c9-689faaf7dda7" containerName="revoke-keys" Oct 30 20:10:54.175576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175572 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0512083-698e-4061-b1c9-689faaf7dda7" containerName="revoke-keys" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175606 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175612 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175627 5733 cpu_manager.go:414] "containerMap: removing container" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175633 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175655 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5c78ece5-ccf1-43ea-9b24-038af64577da" containerName="revoke-keys" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175663 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c78ece5-ccf1-43ea-9b24-038af64577da" containerName="revoke-keys" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175681 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175689 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175709 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175718 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175732 5733 cpu_manager.go:414] "containerMap: removing container" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" Oct 30 20:10:54.175775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175743 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175970 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="058c7110-65ab-4311-9688-b389e30c5f2a" containerName="payload-pre-processing" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.175991 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0512083-698e-4061-b1c9-689faaf7dda7" containerName="revoke-keys" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.176012 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83ebeea-cbdf-47b9-bbfb-8ba86d46ebfe" containerName="payload-pre-processing" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.176025 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc31087c-8882-44ff-9240-42cf43576885" containerName="payload-processing" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.176037 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c78ece5-ccf1-43ea-9b24-038af64577da" containerName="revoke-keys" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.176046 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b964b0d-e172-433f-b5cb-7fd19823d2d7" containerName="payload-processing" Oct 30 20:10:54.176411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.176072 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b80ef9e-12fc-48a1-a912-891358436d57" containerName="payload-pre-processing" Oct 30 20:10:54.213896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.207792 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.213896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.213145 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:54.223939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.223908 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p"] Oct 30 20:10:54.247785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.246154 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhhs\" (UniqueName: \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.247785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.246237 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.247785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.246461 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.276569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.276526 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:54.276737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.276679 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:10:54.348097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.348030 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhhs\" (UniqueName: \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.348297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.348119 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.348297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.348257 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.348879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.348855 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.350541 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.350518 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.359827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.359797 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhhs\" (UniqueName: \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\") pod \"maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.542100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.541992 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:54.851573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:54.851539 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p"] Oct 30 20:10:54.851971 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:10:54.851945 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode190d03a_2abb_4149_aea8_e4d66ed4db7a.slice/crio-fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc WatchSource:0}: Error finding container fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc: Status 404 returned error can't find the container with id fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc Oct 30 20:10:55.125837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:55.125758 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" event={"ID":"e190d03a-2abb-4149-aea8-e4d66ed4db7a","Type":"ContainerStarted","Data":"fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc"} Oct 30 20:10:55.380155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:55.378954 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.136486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.136451 5733 generic.go:356] "Generic (PLEG): container finished" podID="e190d03a-2abb-4149-aea8-e4d66ed4db7a" containerID="01ba6c85150d4e792b44b173c615c9c4679bd1c6cc31132a1676eded0b87bb74" exitCode=0 Oct 30 20:10:56.136650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.136543 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" event={"ID":"e190d03a-2abb-4149-aea8-e4d66ed4db7a","Type":"ContainerDied","Data":"01ba6c85150d4e792b44b173c615c9c4679bd1c6cc31132a1676eded0b87bb74"} Oct 30 20:10:56.150783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.150756 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.520881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.520785 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.728672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.728634 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4"] Oct 30 20:10:56.741661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.741626 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f246b-tzrf4"] Oct 30 20:10:56.796114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.796007 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.946615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:56.946579 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:57.246352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.246319 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:57.338216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.338188 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:57.397576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.397541 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\") pod \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " Oct 30 20:10:57.397833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.397801 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\") pod \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " Oct 30 20:10:57.397931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.397897 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\" (UniqueName: \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\") pod \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\" (UID: \"e190d03a-2abb-4149-aea8-e4d66ed4db7a\") " Oct 30 20:10:57.398144 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.398118 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca" pod "e190d03a-2abb-4149-aea8-e4d66ed4db7a" (UID: "e190d03a-2abb-4149-aea8-e4d66ed4db7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:57.398874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.398847 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:57.400488 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.400431 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs" pod "e190d03a-2abb-4149-aea8-e4d66ed4db7a" (UID: "e190d03a-2abb-4149-aea8-e4d66ed4db7a"). InnerVolumeSpecName "kube-api-access-lkhhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:57.400969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.400945 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca" pod "e190d03a-2abb-4149-aea8-e4d66ed4db7a" (UID: "e190d03a-2abb-4149-aea8-e4d66ed4db7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:57.500129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.500093 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e190d03a-2abb-4149-aea8-e4d66ed4db7a-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:57.500129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:57.500126 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhhs\" (UniqueName: \"kubernetes.io/projected/e190d03a-2abb-4149-aea8-e4d66ed4db7a-kube-api-access-lkhhs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:58.065086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.063961 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0512083-698e-4061-b1c9-689faaf7dda7" path="/var/lib/kubelet/pods/c0512083-698e-4061-b1c9-689faaf7dda7/volumes" Oct 30 20:10:58.070565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.070521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.162355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.162327 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" Oct 30 20:10:58.162510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.162356 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p" event={"ID":"e190d03a-2abb-4149-aea8-e4d66ed4db7a","Type":"ContainerDied","Data":"fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc"} Oct 30 20:10:58.162510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.162401 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc18ed78ce5c1aa8bc9bf0dd8492d0b7d7873e82745e565f99811af775d894fc" Oct 30 20:10:58.865481 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:58.865449 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.390446 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:10:59.390406 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:10:59.571129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.571100 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.767339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.767227 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv"] Oct 30 20:10:59.772981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.772953 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f5eee-dhnlv"] Oct 30 20:10:59.780415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.780391 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.839786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.839752 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf"] Oct 30 20:10:59.840563 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.840545 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e190d03a-2abb-4149-aea8-e4d66ed4db7a" containerName="revoke-keys" Oct 30 20:10:59.840620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.840567 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e190d03a-2abb-4149-aea8-e4d66ed4db7a" containerName="revoke-keys" Oct 30 20:10:59.840742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.840731 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e190d03a-2abb-4149-aea8-e4d66ed4db7a" containerName="revoke-keys" Oct 30 20:10:59.845207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.845187 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:10:59.854095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.851721 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:10:59.854095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.851837 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:10:59.865845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.865810 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf"] Oct 30 20:10:59.928109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.923663 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.943031 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.942993 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:10:59.943211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.943186 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:10:59.943307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:10:59.943276 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfv8w\" (UniqueName: \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.045506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.045402 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.045686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.045534 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.045686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.045575 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfv8w\" (UniqueName: \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.046244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.046215 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.048292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.048264 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.055334 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.055306 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b70387a-2ccb-4660-836b-9111b611df68" path="/var/lib/kubelet/pods/1b70387a-2ccb-4660-836b-9111b611df68/volumes" Oct 30 20:11:00.058244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.058219 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfv8w\" (UniqueName: \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\") pod \"maas-api-revoke-sub-dd32a60d-98nkf\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.191418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.191384 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:00.330139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.329180 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:00.497944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.497911 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf"] Oct 30 20:11:00.498848 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:11:00.498803 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac951dc0_28f6_4afd_a450_ea2adc788134.slice/crio-1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1 WatchSource:0}: Error finding container 1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1: Status 404 returned error can't find the container with id 1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1 Oct 30 20:11:00.802107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:00.801192 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:01.208991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:01.208954 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" event={"ID":"ac951dc0-28f6-4afd-a450-ea2adc788134","Type":"ContainerStarted","Data":"8bca8c7172b58617db212621da6fe28ee99b9b5c7f628cf0e8c1c27660a69fff"} Oct 30 20:11:01.209224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:01.209009 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" event={"ID":"ac951dc0-28f6-4afd-a450-ea2adc788134","Type":"ContainerStarted","Data":"1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1"} Oct 30 20:11:01.232819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:01.232761 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" podStartSLOduration=2.232741569 podStartE2EDuration="2.232741569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:01.226799852 +0000 UTC m=+2421.294635421" watchObservedRunningTime="2026-10-30 20:11:01.232741569 +0000 UTC m=+2421.300577140" Oct 30 20:11:01.652081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:01.643549 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:02.221870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:02.221830 5733 generic.go:356] "Generic (PLEG): container finished" podID="ac951dc0-28f6-4afd-a450-ea2adc788134" containerID="8bca8c7172b58617db212621da6fe28ee99b9b5c7f628cf0e8c1c27660a69fff" exitCode=0 Oct 30 20:11:02.222224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:02.221919 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" event={"ID":"ac951dc0-28f6-4afd-a450-ea2adc788134","Type":"ContainerDied","Data":"8bca8c7172b58617db212621da6fe28ee99b9b5c7f628cf0e8c1c27660a69fff"} Oct 30 20:11:02.520290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:02.520210 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:02.802176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:02.802079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.084164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.084070 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl"] Oct 30 20:11:03.099181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.099149 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-19f2ee-5qcnl"] Oct 30 20:11:03.118364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.118333 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.409401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.409364 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.436865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.436839 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:03.519046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.519015 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\" (UniqueName: \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\") pod \"ac951dc0-28f6-4afd-a450-ea2adc788134\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " Oct 30 20:11:03.519250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.519159 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\") pod \"ac951dc0-28f6-4afd-a450-ea2adc788134\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " Oct 30 20:11:03.519318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.519247 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\") pod \"ac951dc0-28f6-4afd-a450-ea2adc788134\" (UID: \"ac951dc0-28f6-4afd-a450-ea2adc788134\") " Oct 30 20:11:03.519630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.519591 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca" pod "ac951dc0-28f6-4afd-a450-ea2adc788134" (UID: "ac951dc0-28f6-4afd-a450-ea2adc788134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:03.520177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.520150 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:03.521367 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.521336 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w" pod "ac951dc0-28f6-4afd-a450-ea2adc788134" (UID: "ac951dc0-28f6-4afd-a450-ea2adc788134"). InnerVolumeSpecName "kube-api-access-bfv8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:03.522386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.522365 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca" pod "ac951dc0-28f6-4afd-a450-ea2adc788134" (UID: "ac951dc0-28f6-4afd-a450-ea2adc788134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:03.610340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.610302 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.622157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.622121 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac951dc0-28f6-4afd-a450-ea2adc788134-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:03.622157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:03.622153 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfv8w\" (UniqueName: \"kubernetes.io/projected/ac951dc0-28f6-4afd-a450-ea2adc788134-kube-api-access-bfv8w\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:04.054929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.054888 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c78ece5-ccf1-43ea-9b24-038af64577da" path="/var/lib/kubelet/pods/5c78ece5-ccf1-43ea-9b24-038af64577da/volumes" Oct 30 20:11:04.206755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.206722 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:11:04.222797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.222761 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:11:04.260108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.247289 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" Oct 30 20:11:04.260108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.247393 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf" event={"ID":"ac951dc0-28f6-4afd-a450-ea2adc788134","Type":"ContainerDied","Data":"1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1"} Oct 30 20:11:04.260108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.247441 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1167a0ecaf6840c401b2004993868892f835ce7e0c00368b388bf1c5d898f3c1" Oct 30 20:11:04.278943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.278294 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:04.441566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:04.441533 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.152898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.152828 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.379672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.379567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.478720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.478630 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:11:05.479737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.479710 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" containerID="cri-o://ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14" gracePeriod=30 Oct 30 20:11:05.554538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.554489 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.728985 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.728830 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.878650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:05.878617 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:06.589410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:06.589354 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:07.406823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:07.406786 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:07.759525 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:07.759418 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:07.926514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:07.926478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:08.071844 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:08.071758 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:11:08.072010 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:08.071971 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" containerID="cri-o://83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724" gracePeriod=30 Oct 30 20:11:08.172331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:08.172157 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:08.668531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:08.668438 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:09.207297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.207261 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" containerID="cri-o://2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a" gracePeriod=25 Oct 30 20:11:09.225018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.223198 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" containerID="cri-o://6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020" gracePeriod=25 Oct 30 20:11:09.672400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.672199 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:11:09.782605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.779943 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:09.790521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.790489 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:11:09.793414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.793387 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\" (UniqueName: \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\") pod \"460e7d59-f575-457f-9490-e00297324004\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " Oct 30 20:11:09.793597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.793577 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\") pod \"460e7d59-f575-457f-9490-e00297324004\" (UID: \"460e7d59-f575-457f-9490-e00297324004\") " Oct 30 20:11:09.794361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.794330 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume" pod "460e7d59-f575-457f-9490-e00297324004" (UID: "460e7d59-f575-457f-9490-e00297324004"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:09.796723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.796694 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8" pod "460e7d59-f575-457f-9490-e00297324004" (UID: "460e7d59-f575-457f-9490-e00297324004"). InnerVolumeSpecName "kube-api-access-9bpw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:09.896739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.896698 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\" (UniqueName: \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\") pod \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " Oct 30 20:11:09.896963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.896944 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\") pod \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\" (UID: \"1349d220-00d7-44a2-84c8-5b1fafb43ec2\") " Oct 30 20:11:09.897725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.897698 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/460e7d59-f575-457f-9490-e00297324004-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:09.897833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.897729 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpw8\" (UniqueName: \"kubernetes.io/projected/460e7d59-f575-457f-9490-e00297324004-kube-api-access-9bpw8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:09.898228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.898201 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume" pod "1349d220-00d7-44a2-84c8-5b1fafb43ec2" (UID: "1349d220-00d7-44a2-84c8-5b1fafb43ec2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:09.900012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.899989 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj" pod "1349d220-00d7-44a2-84c8-5b1fafb43ec2" (UID: "1349d220-00d7-44a2-84c8-5b1fafb43ec2"). InnerVolumeSpecName "kube-api-access-mvgvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:09.999706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.999671 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgvj\" (UniqueName: \"kubernetes.io/projected/1349d220-00d7-44a2-84c8-5b1fafb43ec2-kube-api-access-mvgvj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:09.999706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:09.999699 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1349d220-00d7-44a2-84c8-5b1fafb43ec2-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:10.160083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.159597 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:10.375714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.375673 5733 generic.go:356] "Generic (PLEG): container finished" podID="460e7d59-f575-457f-9490-e00297324004" containerID="2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a" exitCode=0 Oct 30 20:11:10.376337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.375741 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" Oct 30 20:11:10.376337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.375860 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" event={"ID":"460e7d59-f575-457f-9490-e00297324004","Type":"ContainerDied","Data":"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a"} Oct 30 20:11:10.376337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.375910 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh" event={"ID":"460e7d59-f575-457f-9490-e00297324004","Type":"ContainerDied","Data":"be69da360837380e9d7941f12d5cf36f312a7f3bf20874bf9f25f6d8203f3048"} Oct 30 20:11:10.376337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.375944 5733 scope.go:122] "RemoveContainer" containerID="2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a" Oct 30 20:11:10.379232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.378953 5733 generic.go:356] "Generic (PLEG): container finished" podID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerID="6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020" exitCode=0 Oct 30 20:11:10.379232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.378995 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" event={"ID":"1349d220-00d7-44a2-84c8-5b1fafb43ec2","Type":"ContainerDied","Data":"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020"} Oct 30 20:11:10.379232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.379036 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" event={"ID":"1349d220-00d7-44a2-84c8-5b1fafb43ec2","Type":"ContainerDied","Data":"70168e1c4c60cb038d10d733e78ed6fb2b8184da112522f675e19e308549ddb8"} Oct 30 20:11:10.379232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.379121 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm" Oct 30 20:11:10.392314 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.392294 5733 scope.go:122] "RemoveContainer" containerID="2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a" Oct 30 20:11:10.392590 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:10.392569 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a\": container with ID starting with 2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a not found: ID does not exist" containerID="2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a" Oct 30 20:11:10.392660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.392638 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a"} err="failed to get container status \"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a\": rpc error: code = NotFound desc = could not find container \"2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a\": container with ID starting with 2c73e0517ab5abd336f9c3fb3af588907ad63f3be8572a10e794d9ea68664b8a not found: ID does not exist" Oct 30 20:11:10.392720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.392654 5733 scope.go:122] "RemoveContainer" containerID="6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020" Oct 30 20:11:10.404117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.404090 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:11:10.407664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.407649 5733 scope.go:122] "RemoveContainer" containerID="6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020" Oct 30 20:11:10.408124 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:10.408009 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020\": container with ID starting with 6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020 not found: ID does not exist" containerID="6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020" Oct 30 20:11:10.408124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.408051 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020"} err="failed to get container status \"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020\": rpc error: code = NotFound desc = could not find container \"6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020\": container with ID starting with 6613bae34a9714b28a67639ccd9b64234716087cf671cb1edb07a859acf0d020 not found: ID does not exist" Oct 30 20:11:10.445080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.440608 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-6986a4-54b56f8979-v8qtm"] Oct 30 20:11:10.454813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.454766 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:11:10.462891 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.462858 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6986a4-8c64b6f94-jrvlh"] Oct 30 20:11:10.797289 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.797266 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:11:10.919400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919134 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919189 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919276 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919410 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919459 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919566 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919631 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919764 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.919964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.919939 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\") pod \"f34c469c-95a6-4134-b889-d20cbe59c1a3\" (UID: \"f34c469c-95a6-4134-b889-d20cbe59c1a3\") " Oct 30 20:11:10.920996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.920462 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:10.920996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.920770 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:10.921371 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.921327 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:10.921687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.921649 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:10.921687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.921669 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:10.922219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.922196 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:10.926725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.926700 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "kube-api-access-rc4q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:10.927458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.927435 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:10.929509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.929484 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:10.929700 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:10.929681 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy" pod "f34c469c-95a6-4134-b889-d20cbe59c1a3" (UID: "f34c469c-95a6-4134-b889-d20cbe59c1a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:11.022767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022726 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.022767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022762 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022779 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022795 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022808 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022823 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f34c469c-95a6-4134-b889-d20cbe59c1a3-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022836 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f34c469c-95a6-4134-b889-d20cbe59c1a3-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.023029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.022852 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc4q8\" (UniqueName: \"kubernetes.io/projected/f34c469c-95a6-4134-b889-d20cbe59c1a3-kube-api-access-rc4q8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:11.104701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.104663 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:11.389983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.389946 5733 generic.go:356] "Generic (PLEG): container finished" podID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerID="ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14" exitCode=0 Oct 30 20:11:11.390441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.390004 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" Oct 30 20:11:11.390441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.390042 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerDied","Data":"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14"} Oct 30 20:11:11.390441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.390119 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj" event={"ID":"f34c469c-95a6-4134-b889-d20cbe59c1a3","Type":"ContainerDied","Data":"e7d1da581dc1df5fbc6bbc711ba66233a61b324afb1c6ed06f746a046b7322f2"} Oct 30 20:11:11.390441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.390142 5733 scope.go:122] "RemoveContainer" containerID="ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14" Oct 30 20:11:11.404938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.404920 5733 scope.go:122] "RemoveContainer" containerID="6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3" Oct 30 20:11:11.418925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.418899 5733 scope.go:122] "RemoveContainer" containerID="ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14" Oct 30 20:11:11.419549 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:11.419524 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14\": container with ID starting with ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14 not found: ID does not exist" containerID="ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14" Oct 30 20:11:11.419677 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.419648 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14"} err="failed to get container status \"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14\": rpc error: code = NotFound desc = could not find container \"ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14\": container with ID starting with ddb4ae411941748c611929028a56b4a9ddf50f4c10b9e40f332dd974886d4c14 not found: ID does not exist" Oct 30 20:11:11.419760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.419681 5733 scope.go:122] "RemoveContainer" containerID="6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3" Oct 30 20:11:11.420038 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:11.420012 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3\": container with ID starting with 6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3 not found: ID does not exist" containerID="6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3" Oct 30 20:11:11.420430 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.420049 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3"} err="failed to get container status \"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3\": rpc error: code = NotFound desc = could not find container \"6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3\": container with ID starting with 6a18d8ae0f5b3feb3cce1e00663ac6dd012e00864a08b0c5674d736a8d7b12f3 not found: ID does not exist" Oct 30 20:11:11.422252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.422228 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:11:11.431630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.431605 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f246b-openshift-default-59df6d6865-gt8wj"] Oct 30 20:11:11.960105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:11.958924 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.054825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.054791 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" path="/var/lib/kubelet/pods/1349d220-00d7-44a2-84c8-5b1fafb43ec2/volumes" Oct 30 20:11:12.055260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.055243 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="460e7d59-f575-457f-9490-e00297324004" path="/var/lib/kubelet/pods/460e7d59-f575-457f-9490-e00297324004/volumes" Oct 30 20:11:12.055698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.055684 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" path="/var/lib/kubelet/pods/f34c469c-95a6-4134-b889-d20cbe59c1a3/volumes" Oct 30 20:11:12.165711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.163834 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.309211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.309131 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:11:12.309415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.309383 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" containerID="cri-o://df7fcccb106f728c47213a96c26aea93c8643781123b82544c3584bd03aaa087" gracePeriod=30 Oct 30 20:11:12.398570 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.398526 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b"] Oct 30 20:11:12.400088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400007 5733 cpu_manager.go:414] "containerMap: removing container" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" Oct 30 20:11:12.400249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400233 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" Oct 30 20:11:12.400368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400359 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:12.400432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400423 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:12.400522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400514 5733 cpu_manager.go:414] "containerMap: removing container" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" Oct 30 20:11:12.400595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400584 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" Oct 30 20:11:12.400676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400556 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.400753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400744 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ac951dc0-28f6-4afd-a450-ea2adc788134" containerName="revoke-keys" Oct 30 20:11:12.400818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400810 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac951dc0-28f6-4afd-a450-ea2adc788134" containerName="revoke-keys" Oct 30 20:11:12.400897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400888 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:12.400960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.400952 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:12.401241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.401228 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac951dc0-28f6-4afd-a450-ea2adc788134" containerName="revoke-keys" Oct 30 20:11:12.401335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.401327 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:12.401441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.401429 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="1349d220-00d7-44a2-84c8-5b1fafb43ec2" containerName="payload-pre-processing" Oct 30 20:11:12.401534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.401526 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="460e7d59-f575-457f-9490-e00297324004" containerName="payload-processing" Oct 30 20:11:12.421786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.418799 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.421992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.418700 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b"] Oct 30 20:11:12.450633 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.440302 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:11:12.450633 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.441501 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:12.598231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.581611 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmq5p\" (UniqueName: \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.598231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.581708 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.598231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.582116 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.684921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.684877 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.685159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.685001 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmq5p\" (UniqueName: \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.685159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.685102 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.685965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.685919 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.687780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.687758 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.697457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.697428 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmq5p\" (UniqueName: \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\") pod \"maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:12.779023 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:12.778819 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:13.115135 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.114342 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b"] Oct 30 20:11:13.115135 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:11:13.114646 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2469df15_3dd0_4c4b_bf49_d0d8ac187c1f.slice/crio-121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150 WatchSource:0}: Error finding container 121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150: Status 404 returned error can't find the container with id 121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150 Oct 30 20:11:13.408743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.408624 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:11:13.505409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.505362 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" event={"ID":"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f","Type":"ContainerStarted","Data":"121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150"} Oct 30 20:11:13.506841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.506815 5733 generic.go:356] "Generic (PLEG): container finished" podID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerID="83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724" exitCode=0 Oct 30 20:11:13.506995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.506868 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" Oct 30 20:11:13.506995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.506902 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" event={"ID":"fa7be5f2-303b-41de-972c-4a36f84f127d","Type":"ContainerDied","Data":"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724"} Oct 30 20:11:13.506995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.506952 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h" event={"ID":"fa7be5f2-303b-41de-972c-4a36f84f127d","Type":"ContainerDied","Data":"f866320ed1746d019209666b6fad0bc0d9110c8b74fd5354227143ecebb5110b"} Oct 30 20:11:13.506995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.506982 5733 scope.go:122] "RemoveContainer" containerID="83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724" Oct 30 20:11:13.513489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513468 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513531 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513577 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513641 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513745 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513789 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.513838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513807 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.514219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513929 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.514219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.513988 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\") pod \"fa7be5f2-303b-41de-972c-4a36f84f127d\" (UID: \"fa7be5f2-303b-41de-972c-4a36f84f127d\") " Oct 30 20:11:13.520148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.520123 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:13.520347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.520323 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:13.520435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.520406 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:13.520888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.520864 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:13.521142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.521109 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:13.522361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.522337 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:13.522769 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.522747 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "kube-api-access-z4lmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:13.522918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.522892 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:13.523330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.523309 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy" pod "fa7be5f2-303b-41de-972c-4a36f84f127d" (UID: "fa7be5f2-303b-41de-972c-4a36f84f127d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:13.535619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.535597 5733 scope.go:122] "RemoveContainer" containerID="83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724" Oct 30 20:11:13.535969 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:13.535946 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724\": container with ID starting with 83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724 not found: ID does not exist" containerID="83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724" Oct 30 20:11:13.536086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.535979 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724"} err="failed to get container status \"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724\": rpc error: code = NotFound desc = could not find container \"83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724\": container with ID starting with 83fe5572658b68dc46861b83d55aa01fab885292ed74cd794dde9172c441b724 not found: ID does not exist" Oct 30 20:11:13.615817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615783 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.615817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615820 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615836 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615849 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa7be5f2-303b-41de-972c-4a36f84f127d-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615864 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4lmb\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-kube-api-access-z4lmb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615878 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615891 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615905 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.625701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.615919 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa7be5f2-303b-41de-972c-4a36f84f127d-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:13.687602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.687517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:13.844601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.844568 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:11:13.853594 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:13.853558 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1f5eee-openshift-default-8856f869c-gj77h"] Oct 30 20:11:14.057733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:14.057656 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" path="/var/lib/kubelet/pods/fa7be5f2-303b-41de-972c-4a36f84f127d/volumes" Oct 30 20:11:14.097718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:14.097643 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:14.520881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:14.519503 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" event={"ID":"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f","Type":"ContainerStarted","Data":"72056619d34752340c8c952489caf9900bc7c79d2467d9b793fb98320192ba41"} Oct 30 20:11:14.550244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:14.549735 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" podStartSLOduration=2.5497157379999997 podStartE2EDuration="2.549715738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:14.544629717 +0000 UTC m=+2434.612465287" watchObservedRunningTime="2026-10-30 20:11:14.549715738 +0000 UTC m=+2434.617551308" Oct 30 20:11:15.102723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:15.102671 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:15.567162 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:15.567025 5733 generic.go:356] "Generic (PLEG): container finished" podID="2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" containerID="72056619d34752340c8c952489caf9900bc7c79d2467d9b793fb98320192ba41" exitCode=0 Oct 30 20:11:15.568737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:15.567355 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" event={"ID":"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f","Type":"ContainerDied","Data":"72056619d34752340c8c952489caf9900bc7c79d2467d9b793fb98320192ba41"} Oct 30 20:11:16.250567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.250528 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:16.481634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.481558 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:16.614083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.610449 5733 generic.go:356] "Generic (PLEG): container finished" podID="63919396-52a0-4723-9d83-dd3849acdbc1" containerID="5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa" exitCode=0 Oct 30 20:11:16.614083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.610797 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerDied","Data":"5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa"} Oct 30 20:11:16.614083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.610851 5733 scope.go:122] "RemoveContainer" containerID="174137be6cb2f13be5c30546e48c66ee0b0c6139b4848b7fe4d1076cebb55397" Oct 30 20:11:16.614083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.611398 5733 scope.go:122] "RemoveContainer" containerID="5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa" Oct 30 20:11:16.614083 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:16.611841 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j_openshift-ingress(63919396-52a0-4723-9d83-dd3849acdbc1)\"" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" Oct 30 20:11:16.867593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:16.867022 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:17.064475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.063663 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:17.110116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.104523 5733 patch_prober.go:29] interesting pod/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:11:17.110116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.104602 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:11:17.239104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.238969 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\") pod \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " Oct 30 20:11:17.239262 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.239182 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\") pod \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " Oct 30 20:11:17.239311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.239260 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\" (UniqueName: \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\") pod \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\" (UID: \"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f\") " Oct 30 20:11:17.239763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.239732 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca" pod "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" (UID: "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:17.239940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.239921 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.241807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.241783 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p" pod "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" (UID: "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f"). InnerVolumeSpecName "kube-api-access-jmq5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:17.242034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.242010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca" pod "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" (UID: "2469df15-3dd0-4c4b-bf49-d0d8ac187c1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:17.343766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.343723 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.343766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.343758 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmq5p\" (UniqueName: \"kubernetes.io/projected/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f-kube-api-access-jmq5p\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.672111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.671871 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:11:17.674503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.674254 5733 generic.go:356] "Generic (PLEG): container finished" podID="46e31201-64ff-4e20-999c-7e3be595e84a" containerID="df7fcccb106f728c47213a96c26aea93c8643781123b82544c3584bd03aaa087" exitCode=0 Oct 30 20:11:17.674503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.674375 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerDied","Data":"df7fcccb106f728c47213a96c26aea93c8643781123b82544c3584bd03aaa087"} Oct 30 20:11:17.674503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.674433 5733 scope.go:122] "RemoveContainer" containerID="df7fcccb106f728c47213a96c26aea93c8643781123b82544c3584bd03aaa087" Oct 30 20:11:17.677925 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.677904 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" event={"ID":"2469df15-3dd0-4c4b-bf49-d0d8ac187c1f","Type":"ContainerDied","Data":"121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150"} Oct 30 20:11:17.678092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.677940 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="121e96fd8bece0e429832d6d744b5021e8f20f3a42a7039296edd801a6790150" Oct 30 20:11:17.678092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.677982 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b" Oct 30 20:11:17.694344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.694318 5733 scope.go:122] "RemoveContainer" containerID="b4f1127e96b1587f58efc3755a137dc8b764092b74c96e0e4bcfd26794c0cd4e" Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867310 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867461 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867513 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867586 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867752 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867814 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867877 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867910 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.869153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.867980 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\") pod \"46e31201-64ff-4e20-999c-7e3be595e84a\" (UID: \"46e31201-64ff-4e20-999c-7e3be595e84a\") " Oct 30 20:11:17.870763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.870036 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:17.870763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.870262 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:17.870763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.870574 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:17.870763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.870592 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:17.871557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.871407 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:17.871819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.871786 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:17.871930 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.871901 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:17.872597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.872573 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:17.874373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.874338 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q" pod "46e31201-64ff-4e20-999c-7e3be595e84a" (UID: "46e31201-64ff-4e20-999c-7e3be595e84a"). InnerVolumeSpecName "kube-api-access-cnx6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971470 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e31201-64ff-4e20-999c-7e3be595e84a-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971511 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971524 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971539 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnx6q\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-kube-api-access-cnx6q\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971556 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e31201-64ff-4e20-999c-7e3be595e84a-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971570 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e31201-64ff-4e20-999c-7e3be595e84a-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971585 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971598 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:17.972191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:17.971612 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e31201-64ff-4e20-999c-7e3be595e84a-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:18.028613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.028572 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:18.660743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.660706 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:11:18.679202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.679167 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:11:18.693795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.693763 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" event={"ID":"46e31201-64ff-4e20-999c-7e3be595e84a","Type":"ContainerDied","Data":"4f0bac2ba5fa50c8f237a50bdfdd15b6d1457f8740cd28335adbf774aaea3cc7"} Oct 30 20:11:18.693972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.693909 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc" Oct 30 20:11:18.777877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.777843 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:11:18.787655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.785543 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-19f2ee-openshift-default-786d8c949c-v4rwc"] Oct 30 20:11:18.802313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.802283 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:18.832081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.826143 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:11:18.832081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:18.826795 5733 scope.go:122] "RemoveContainer" containerID="5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa" Oct 30 20:11:18.832081 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:18.827236 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"istio-proxy\" with CrashLoopBackOff: \"back-off 10s restarting failed container=istio-proxy pod=e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j_openshift-ingress(63919396-52a0-4723-9d83-dd3849acdbc1)\"" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" Oct 30 20:11:19.030117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:19.030004 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:19.266947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:19.266827 5733 scope.go:122] "RemoveContainer" containerID="02848a84625fbfe843ce6f0547219ecfb97d6cc22d83d4d4acf1da704bb031b9" Oct 30 20:11:19.284290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:19.282991 5733 scope.go:122] "RemoveContainer" containerID="69362b96857b4c7a07afab4032a9a6d2bd08830fa2ad6d8ea620dfe29728ba98" Oct 30 20:11:19.382229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:19.382199 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.055614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:20.055577 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" path="/var/lib/kubelet/pods/46e31201-64ff-4e20-999c-7e3be595e84a/volumes" Oct 30 20:11:20.482933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:20.482884 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.742201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:20.733356 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:21.119310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:21.119272 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:21.119862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:21.119844 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p"] Oct 30 20:11:21.124540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:21.124515 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6986a4-mcg5p"] Oct 30 20:11:21.268773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:21.268449 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:21.684954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:21.684918 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.056423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:22.056337 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e190d03a-2abb-4149-aea8-e4d66ed4db7a" path="/var/lib/kubelet/pods/e190d03a-2abb-4149-aea8-e4d66ed4db7a/volumes" Oct 30 20:11:22.088302 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:22.088267 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.718277 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:22.718232 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.880926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:22.880890 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.050261 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:23.048046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.661260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:23.661219 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" containerID="cri-o://82879374448bffed3401cd51fab5da854aeef69148f557bff724528cc4ffa351" gracePeriod=25 Oct 30 20:11:23.679770 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:23.679731 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" containerID="cri-o://5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5" gracePeriod=25 Oct 30 20:11:23.790350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:23.790309 5733 generic.go:356] "Generic (PLEG): container finished" podID="229609ef-e005-41c3-807a-54fa69b874d9" containerID="82879374448bffed3401cd51fab5da854aeef69148f557bff724528cc4ffa351" exitCode=0 Oct 30 20:11:23.790768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:23.790382 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" event={"ID":"229609ef-e005-41c3-807a-54fa69b874d9","Type":"ContainerDied","Data":"82879374448bffed3401cd51fab5da854aeef69148f557bff724528cc4ffa351"} Oct 30 20:11:24.019109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.016437 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:24.039554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.039524 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:11:24.124076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.123421 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\" (UniqueName: \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\") pod \"229609ef-e005-41c3-807a-54fa69b874d9\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " Oct 30 20:11:24.124076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.123568 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\") pod \"229609ef-e005-41c3-807a-54fa69b874d9\" (UID: \"229609ef-e005-41c3-807a-54fa69b874d9\") " Oct 30 20:11:24.125462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.125428 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume" pod "229609ef-e005-41c3-807a-54fa69b874d9" (UID: "229609ef-e005-41c3-807a-54fa69b874d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:24.130754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.128440 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967" pod "229609ef-e005-41c3-807a-54fa69b874d9" (UID: "229609ef-e005-41c3-807a-54fa69b874d9"). InnerVolumeSpecName "kube-api-access-lr967". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:24.138813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.138436 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:11:24.225198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.224926 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:24.226410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.226377 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr967\" (UniqueName: \"kubernetes.io/projected/229609ef-e005-41c3-807a-54fa69b874d9-kube-api-access-lr967\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:24.226569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.226546 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229609ef-e005-41c3-807a-54fa69b874d9-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:24.327545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.327498 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\") pod \"6862aa9e-2401-4f55-b63d-25435bf3eac4\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " Oct 30 20:11:24.327847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.327832 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\" (UniqueName: \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\") pod \"6862aa9e-2401-4f55-b63d-25435bf3eac4\" (UID: \"6862aa9e-2401-4f55-b63d-25435bf3eac4\") " Oct 30 20:11:24.328434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.327990 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume" pod "6862aa9e-2401-4f55-b63d-25435bf3eac4" (UID: "6862aa9e-2401-4f55-b63d-25435bf3eac4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:24.329271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.329250 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6862aa9e-2401-4f55-b63d-25435bf3eac4-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:24.332860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.332835 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx" pod "6862aa9e-2401-4f55-b63d-25435bf3eac4" (UID: "6862aa9e-2401-4f55-b63d-25435bf3eac4"). InnerVolumeSpecName "kube-api-access-chjjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:24.432930 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.432892 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chjjx\" (UniqueName: \"kubernetes.io/projected/6862aa9e-2401-4f55-b63d-25435bf3eac4-kube-api-access-chjjx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:24.802902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.802870 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" Oct 30 20:11:24.803425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.802904 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl" event={"ID":"229609ef-e005-41c3-807a-54fa69b874d9","Type":"ContainerDied","Data":"e4807336b1f3448b427f98057c91236861fdce26279d0ea67e305c5adc8df3f4"} Oct 30 20:11:24.803425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.802966 5733 scope.go:122] "RemoveContainer" containerID="82879374448bffed3401cd51fab5da854aeef69148f557bff724528cc4ffa351" Oct 30 20:11:24.805238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.805208 5733 generic.go:356] "Generic (PLEG): container finished" podID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerID="5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5" exitCode=0 Oct 30 20:11:24.805469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.805446 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" event={"ID":"6862aa9e-2401-4f55-b63d-25435bf3eac4","Type":"ContainerDied","Data":"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5"} Oct 30 20:11:24.805542 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.805501 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" event={"ID":"6862aa9e-2401-4f55-b63d-25435bf3eac4","Type":"ContainerDied","Data":"7cd4d270288634355f9503d201a1f2cfdc519dc7193da005df07ae91e6a86bd8"} Oct 30 20:11:24.805601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.805582 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl" Oct 30 20:11:24.822446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.822421 5733 scope.go:122] "RemoveContainer" containerID="5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5" Oct 30 20:11:24.839453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.839434 5733 scope.go:122] "RemoveContainer" containerID="5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5" Oct 30 20:11:24.839821 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:24.839791 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5\": container with ID starting with 5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5 not found: ID does not exist" containerID="5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5" Oct 30 20:11:24.839894 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.839833 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5"} err="failed to get container status \"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5\": rpc error: code = NotFound desc = could not find container \"5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5\": container with ID starting with 5a6753e5d38dcdf01b5aa3eed214b312fd9754637878ed468f54de905a76ddf5 not found: ID does not exist" Oct 30 20:11:24.876265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.860788 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:11:24.876265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.865634 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d2e0-84c9b5c6d4-q97pl"] Oct 30 20:11:24.881493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.881468 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:11:24.888038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:24.888015 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-55d2e0-f94db459f-9xhgl"] Oct 30 20:11:25.393967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.393933 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.818672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.818587 5733 generic.go:356] "Generic (PLEG): container finished" podID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerID="0d226d4f4313ad8b338e662dc99348dd946b857c1cedcac66ea275853e13df7d" exitCode=0 Oct 30 20:11:25.819166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.818665 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerDied","Data":"0d226d4f4313ad8b338e662dc99348dd946b857c1cedcac66ea275853e13df7d"} Oct 30 20:11:25.819259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.819216 5733 scope.go:122] "RemoveContainer" containerID="0d226d4f4313ad8b338e662dc99348dd946b857c1cedcac66ea275853e13df7d" Oct 30 20:11:25.821509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.821470 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:25.821632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.821561 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:25.821632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:25.821611 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:26.052589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:26.052551 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229609ef-e005-41c3-807a-54fa69b874d9" path="/var/lib/kubelet/pods/229609ef-e005-41c3-807a-54fa69b874d9/volumes" Oct 30 20:11:26.053126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:26.053106 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" path="/var/lib/kubelet/pods/6862aa9e-2401-4f55-b63d-25435bf3eac4/volumes" Oct 30 20:11:26.391438 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:26.391409 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:26.852093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:26.852039 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerStarted","Data":"f343c6aecce68c616c96736e712eda12461cff3ea6b0d1d37c859d06999ee229"} Oct 30 20:11:27.043379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.043337 5733 scope.go:122] "RemoveContainer" containerID="5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa" Oct 30 20:11:27.045354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.045317 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:27.045517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.045407 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:27.045517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.045460 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:11:27.063036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.063006 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:27.063412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.063384 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:27.063521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.063432 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:27.375342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.375304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.866978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:27.866939 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerStarted","Data":"7443325ce499f938226570f10ec3bbdf5f4ac6b2926a209de3408743b89991d2"} Oct 30 20:11:28.078254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.078200 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:28.078468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.078283 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:28.255601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.255494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:28.543143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.542950 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:28.704891 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.704854 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:28.789398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.789364 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:11:28.790659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.790632 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 20:11:28.790853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.790834 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 20:11:28.909317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.909280 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:11:28.928087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:28.920911 5733 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" secret="" err="secret \"e2e-inf-b-6986a4-openshift-default-dockercfg-rwt7k\" not found" Oct 30 20:11:29.114271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.114235 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:29.114635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.114603 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:29.117214 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.117190 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.268143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.265970 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.491618 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:29.491581 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:11:29.504370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.504343 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.677437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.677394 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.791255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.791222 5733 patch_prober.go:29] interesting pod/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 20:11:29.791534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.791505 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 20:11:29.981135 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:29.971958 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" containerID="cri-o://7443325ce499f938226570f10ec3bbdf5f4ac6b2926a209de3408743b89991d2" gracePeriod=30 Oct 30 20:11:30.086139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:30.083352 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:30.086139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:30.083406 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:30.662315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:30.662275 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:30.812071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:30.812009 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.063552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:31.063454 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:31.064149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:31.064114 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:31.916180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:31.916146 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:32.063692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:32.063654 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 20:11:32.095733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:32.095687 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 20:11:32.163082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:32.162655 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.068147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.068116 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:33.069331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.069309 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:33.108514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.108484 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.763963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.763927 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.824308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.824278 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:11:33.916450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:33.916416 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.072710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.072626 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.397634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.397598 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.503207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.503175 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:11:34.521394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.521357 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 20:11:34.546299 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.546263 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 20:11:34.560001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.559972 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 20:11:34.688791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.688713 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.864287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:34.864249 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.041965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:35.037835 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.971916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:35.971882 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.131699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:36.131663 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.586052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:36.585993 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.834117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:36.834084 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:11:36.834600 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:36.834567 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" containerID="cri-o://f49301dd7eca1a9597e87b1bbfbfef3375a3bcf03f43d03b612f196352da746f" gracePeriod=30 Oct 30 20:11:37.002045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.002014 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:37.183121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.171984 5733 generic.go:356] "Generic (PLEG): container finished" podID="21b6051c-d265-4cd2-b08b-b64054880109" containerID="f49301dd7eca1a9597e87b1bbfbfef3375a3bcf03f43d03b612f196352da746f" exitCode=0 Oct 30 20:11:37.183121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.172233 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerDied","Data":"f49301dd7eca1a9597e87b1bbfbfef3375a3bcf03f43d03b612f196352da746f"} Oct 30 20:11:37.183121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.172286 5733 scope.go:122] "RemoveContainer" containerID="3b5938b920c46305844ab20b3bdf34ca444b5127542d08db12250f2b7377a905" Oct 30 20:11:37.192187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.187041 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:37.273337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.273311 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:11:37.346047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.346007 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\") pod \"21b6051c-d265-4cd2-b08b-b64054880109\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " Oct 30 20:11:37.346274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.346090 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\") pod \"21b6051c-d265-4cd2-b08b-b64054880109\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " Oct 30 20:11:37.346358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.346324 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\") pod \"21b6051c-d265-4cd2-b08b-b64054880109\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " Oct 30 20:11:37.346530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.346500 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca" pod "21b6051c-d265-4cd2-b08b-b64054880109" (UID: "21b6051c-d265-4cd2-b08b-b64054880109"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:37.346761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.346738 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\" (UniqueName: \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\") pod \"21b6051c-d265-4cd2-b08b-b64054880109\" (UID: \"21b6051c-d265-4cd2-b08b-b64054880109\") " Oct 30 20:11:37.347655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.347627 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/21b6051c-d265-4cd2-b08b-b64054880109-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:37.349403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.349366 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs" pod "21b6051c-d265-4cd2-b08b-b64054880109" (UID: "21b6051c-d265-4cd2-b08b-b64054880109"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:37.349685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.349651 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm" pod "21b6051c-d265-4cd2-b08b-b64054880109" (UID: "21b6051c-d265-4cd2-b08b-b64054880109"). InnerVolumeSpecName "kube-api-access-hj2vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:37.349947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.349924 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls" pod "21b6051c-d265-4cd2-b08b-b64054880109" (UID: "21b6051c-d265-4cd2-b08b-b64054880109"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:37.404273 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.404237 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:37.448924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.448845 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:37.448924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.448876 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2vm\" (UniqueName: \"kubernetes.io/projected/21b6051c-d265-4cd2-b08b-b64054880109-kube-api-access-hj2vm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:37.448924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.448890 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/21b6051c-d265-4cd2-b08b-b64054880109-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:37.551074 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.550922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:37.992692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:37.992658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:38.181088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.181031 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" Oct 30 20:11:38.181546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.181100 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2" event={"ID":"21b6051c-d265-4cd2-b08b-b64054880109","Type":"ContainerDied","Data":"0f5028e2c38b8218ca934754132cac5f323b64978c15727a8df381d5058310a2"} Oct 30 20:11:38.181546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.181167 5733 scope.go:122] "RemoveContainer" containerID="f49301dd7eca1a9597e87b1bbfbfef3375a3bcf03f43d03b612f196352da746f" Oct 30 20:11:38.208615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.208590 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:11:38.213369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.213348 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a7bb6c-6ff4687775-26zw2"] Oct 30 20:11:38.698798 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:38.698760 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:39.193108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.193050 5733 generic.go:356] "Generic (PLEG): container finished" podID="63919396-52a0-4723-9d83-dd3849acdbc1" containerID="7443325ce499f938226570f10ec3bbdf5f4ac6b2926a209de3408743b89991d2" exitCode=0 Oct 30 20:11:39.193605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.193132 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerDied","Data":"7443325ce499f938226570f10ec3bbdf5f4ac6b2926a209de3408743b89991d2"} Oct 30 20:11:39.193605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.193216 5733 scope.go:122] "RemoveContainer" containerID="5d69ca81162388df7c27228240be91e3f4cb1f159477b73895eaca6762dde9aa" Oct 30 20:11:39.406356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.406330 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483285 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483345 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483411 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483460 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483520 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483704 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483763 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483836 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.488167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.483865 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\") pod \"63919396-52a0-4723-9d83-dd3849acdbc1\" (UID: \"63919396-52a0-4723-9d83-dd3849acdbc1\") " Oct 30 20:11:39.491867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.489880 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:39.491867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.490626 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:39.492016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.491924 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:39.492462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.492395 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:39.492813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.492710 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:39.492936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.492874 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:39.493539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.493487 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:39.503698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.503531 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" containerID="cri-o://b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586" gracePeriod=25 Oct 30 20:11:39.503698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.503612 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "kube-api-access-7ls6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:39.506140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.504751 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:39.512279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.512248 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy" pod "63919396-52a0-4723-9d83-dd3849acdbc1" (UID: "63919396-52a0-4723-9d83-dd3849acdbc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:39.590887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590837 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.590887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590883 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ls6g\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-kube-api-access-7ls6g\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.590887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590894 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590904 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590914 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590925 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63919396-52a0-4723-9d83-dd3849acdbc1-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590934 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63919396-52a0-4723-9d83-dd3849acdbc1-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590943 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63919396-52a0-4723-9d83-dd3849acdbc1-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.591154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.590952 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63919396-52a0-4723-9d83-dd3849acdbc1-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.787070 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.787034 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:11:39.801678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.801642 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 20:11:39.836889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.836854 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 20:11:39.896116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.894664 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\") pod \"580aade9-3f41-41ff-a977-4a10a7814421\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " Oct 30 20:11:39.896116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.894837 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\" (UniqueName: \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\") pod \"580aade9-3f41-41ff-a977-4a10a7814421\" (UID: \"580aade9-3f41-41ff-a977-4a10a7814421\") " Oct 30 20:11:39.896116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.895135 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume" pod "580aade9-3f41-41ff-a977-4a10a7814421" (UID: "580aade9-3f41-41ff-a977-4a10a7814421"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:39.896116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.895551 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/580aade9-3f41-41ff-a977-4a10a7814421-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:39.901943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.901911 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk" pod "580aade9-3f41-41ff-a977-4a10a7814421" (UID: "580aade9-3f41-41ff-a977-4a10a7814421"). InnerVolumeSpecName "kube-api-access-x2rlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:39.958742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.958692 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:39.996593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:39.996562 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rlk\" (UniqueName: \"kubernetes.io/projected/580aade9-3f41-41ff-a977-4a10a7814421-kube-api-access-x2rlk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.053946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.053871 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b6051c-d265-4cd2-b08b-b64054880109" path="/var/lib/kubelet/pods/21b6051c-d265-4cd2-b08b-b64054880109/volumes" Oct 30 20:11:40.102267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.102236 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.228642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.228603 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" event={"ID":"63919396-52a0-4723-9d83-dd3849acdbc1","Type":"ContainerDied","Data":"fbb8669e9aa14cf72ece6b925664a4340f2cccda92ec8c05fab95d372e6f80a6"} Oct 30 20:11:40.229220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.228681 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j" Oct 30 20:11:40.229220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.228803 5733 scope.go:122] "RemoveContainer" containerID="7443325ce499f938226570f10ec3bbdf5f4ac6b2926a209de3408743b89991d2" Oct 30 20:11:40.231892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.231868 5733 generic.go:356] "Generic (PLEG): container finished" podID="580aade9-3f41-41ff-a977-4a10a7814421" containerID="b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586" exitCode=0 Oct 30 20:11:40.232019 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.231929 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" Oct 30 20:11:40.232166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.232093 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" event={"ID":"580aade9-3f41-41ff-a977-4a10a7814421","Type":"ContainerDied","Data":"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586"} Oct 30 20:11:40.232166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.232141 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb" event={"ID":"580aade9-3f41-41ff-a977-4a10a7814421","Type":"ContainerDied","Data":"33729930ee1c1cdda6e6217164e804946c74a9778d4112a0ee5b2e07719e73ed"} Oct 30 20:11:40.249151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.248659 5733 scope.go:122] "RemoveContainer" containerID="b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586" Oct 30 20:11:40.258107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.257978 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:11:40.262112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.262089 5733 scope.go:122] "RemoveContainer" containerID="b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586" Oct 30 20:11:40.262468 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:40.262447 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586\": container with ID starting with b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586 not found: ID does not exist" containerID="b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586" Oct 30 20:11:40.262545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.262482 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586"} err="failed to get container status \"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586\": rpc error: code = NotFound desc = could not find container \"b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586\": container with ID starting with b5e5dbde170b2e860c545b1bda2172b66de27c9e1f2f9bbf726ec39680375586 not found: ID does not exist" Oct 30 20:11:40.262628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.262611 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6986a4-openshift-default-68b58c765-7vc8j"] Oct 30 20:11:40.273880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.272746 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:11:40.277679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.277653 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a7bb6c-6c94759ccd-9h5kb"] Oct 30 20:11:40.904757 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:40.904724 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.133750 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:41.133710 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.289038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:41.288951 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:41.986352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:41.986324 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.075456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.075419 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="580aade9-3f41-41ff-a977-4a10a7814421" path="/var/lib/kubelet/pods/580aade9-3f41-41ff-a977-4a10a7814421/volumes" Oct 30 20:11:42.076049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.076027 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" path="/var/lib/kubelet/pods/63919396-52a0-4723-9d83-dd3849acdbc1/volumes" Oct 30 20:11:42.173755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.173720 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.325579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.325496 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:11:42.326031 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.325747 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" containerID="cri-o://df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135" gracePeriod=30 Oct 30 20:11:42.412984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:42.412891 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.012673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.012645 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.168089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.167795 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.313136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.313039 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.727699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.727667 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b"] Oct 30 20:11:43.741979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.741948 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a4f02-rck4b"] Oct 30 20:11:43.831475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:43.831438 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:44.051605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.051526 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" path="/var/lib/kubelet/pods/2469df15-3dd0-4c4b-bf49-d0d8ac187c1f/volumes" Oct 30 20:11:44.052908 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.052875 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:44.126305 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.126272 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:11:44.126528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.126484 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" containerID="cri-o://f343c6aecce68c616c96736e712eda12461cff3ea6b0d1d37c859d06999ee229" gracePeriod=30 Oct 30 20:11:44.207372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.207344 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:44.350504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.350472 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:44.802989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.802897 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" containerID="cri-o://7811792985c4220199a6babde04642377e65c376b6378e9c11f93f518cffa9ab" gracePeriod=25 Oct 30 20:11:44.837547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:44.837508 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" containerID="cri-o://1edfb42b38ed41db97908a3580b4546b999ed0c797f4ab555f90a0f9861ebf85" gracePeriod=25 Oct 30 20:11:45.206384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:45.206351 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:45.880357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:45.880320 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:46.023103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:46.023066 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:46.301879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:46.301788 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:46.493859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:46.493823 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:47.032706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.032663 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:47.614253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.614215 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:47.659889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.659857 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:11:47.836417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836383 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836443 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836483 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836568 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836606 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836659 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.836902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836817 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:47.836902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836877 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.837128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836927 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.837128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.836982 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\") pod \"d421fe44-7234-405c-a2fc-e21240a4f05d\" (UID: \"d421fe44-7234-405c-a2fc-e21240a4f05d\") " Oct 30 20:11:47.837244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837107 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:47.837408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837388 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:47.837717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837684 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:47.837918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837884 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:47.837918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837827 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d421fe44-7234-405c-a2fc-e21240a4f05d-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.838204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837935 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.838204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837953 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.838204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.837969 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.841043 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.841010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:47.841270 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.841224 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "kube-api-access-wxdwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:47.843611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.843586 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:47.844203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.844178 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token" pod "d421fe44-7234-405c-a2fc-e21240a4f05d" (UID: "d421fe44-7234-405c-a2fc-e21240a4f05d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:47.935724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.935639 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 20:11:47.939899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.939770 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.939899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.939808 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.939899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.939825 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.939899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.939839 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d421fe44-7234-405c-a2fc-e21240a4f05d-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:47.939899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:47.939855 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxdwt\" (UniqueName: \"kubernetes.io/projected/d421fe44-7234-405c-a2fc-e21240a4f05d-kube-api-access-wxdwt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:48.068947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.068904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:48.075294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.075232 5733 patch_prober.go:29] interesting pod/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:11:48.075452 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.075292 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:11:48.311324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.311241 5733 generic.go:356] "Generic (PLEG): container finished" podID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerID="df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135" exitCode=0 Oct 30 20:11:48.311324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.311281 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" event={"ID":"d421fe44-7234-405c-a2fc-e21240a4f05d","Type":"ContainerDied","Data":"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135"} Oct 30 20:11:48.311324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.311294 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" Oct 30 20:11:48.311324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.311321 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf" event={"ID":"d421fe44-7234-405c-a2fc-e21240a4f05d","Type":"ContainerDied","Data":"42a9fff551b9201bf2ca7fb1effb3afbe9e76099086a8d15663b1fbf3c911e30"} Oct 30 20:11:48.311631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.311342 5733 scope.go:122] "RemoveContainer" containerID="df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135" Oct 30 20:11:48.324125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.324106 5733 scope.go:122] "RemoveContainer" containerID="df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135" Oct 30 20:11:48.324493 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:48.324465 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135\": container with ID starting with df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135 not found: ID does not exist" containerID="df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135" Oct 30 20:11:48.324575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.324506 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135"} err="failed to get container status \"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135\": rpc error: code = NotFound desc = could not find container \"df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135\": container with ID starting with df4e21932d415b1481fb159eaad2af7520f5caf5ade8f639188dfdc2d228a135 not found: ID does not exist" Oct 30 20:11:48.335151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.335121 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:11:48.339984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.339956 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-db02a7-openshift-default-7dc8dd5f97-lgqkf"] Oct 30 20:11:48.808766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:48.806359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:49.339414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.339375 5733 generic.go:356] "Generic (PLEG): container finished" podID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerID="f343c6aecce68c616c96736e712eda12461cff3ea6b0d1d37c859d06999ee229" exitCode=0 Oct 30 20:11:49.339906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.339450 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerDied","Data":"f343c6aecce68c616c96736e712eda12461cff3ea6b0d1d37c859d06999ee229"} Oct 30 20:11:49.339906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.339520 5733 scope.go:122] "RemoveContainer" containerID="0d226d4f4313ad8b338e662dc99348dd946b857c1cedcac66ea275853e13df7d" Oct 30 20:11:49.435584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.435559 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:49.562530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562491 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562588 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562627 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562709 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562776 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562814 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.562909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562813 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:49.563085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.562978 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.563085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563012 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.563085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563049 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\") pod \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\" (UID: \"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac\") " Oct 30 20:11:49.563282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563255 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:49.563597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563570 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:49.563758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563617 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:49.563932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563878 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:49.563932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563903 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.564195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563941 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.564195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563956 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.564195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.563968 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.565199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.565172 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "kube-api-access-7dr2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:49.565345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.565325 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:49.565791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.565772 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:49.566267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.566242 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token" pod "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" (UID: "8b1adab2-ae8a-4b80-9273-97fb0d04e5ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:49.595320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.595250 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" containerID="cri-o://940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c" gracePeriod=45 Oct 30 20:11:49.596567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.596519 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" containerID="cri-o://10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9" gracePeriod=45 Oct 30 20:11:49.617568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.617522 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" containerID="cri-o://5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0" gracePeriod=45 Oct 30 20:11:49.650997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.650963 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652026 5733 cpu_manager.go:414] "containerMap: removing container" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652069 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652096 5733 cpu_manager.go:414] "containerMap: removing container" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652105 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652132 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652141 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652158 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652166 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652187 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652196 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652209 5733 cpu_manager.go:414] "containerMap: removing container" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652217 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652236 5733 cpu_manager.go:414] "containerMap: removing container" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652243 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.652052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652264 5733 cpu_manager.go:414] "containerMap: removing container" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652272 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652293 5733 cpu_manager.go:414] "containerMap: removing container" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652301 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652313 5733 cpu_manager.go:414] "containerMap: removing container" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652321 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652336 5733 cpu_manager.go:414] "containerMap: removing container" podUID="2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" containerName="revoke-keys" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652343 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" containerName="revoke-keys" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652362 5733 cpu_manager.go:414] "containerMap: removing container" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652369 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652392 5733 cpu_manager.go:414] "containerMap: removing container" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652399 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652556 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652570 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652580 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652594 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652603 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="2469df15-3dd0-4c4b-bf49-d0d8ac187c1f" containerName="revoke-keys" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652615 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652625 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="63919396-52a0-4723-9d83-dd3849acdbc1" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652634 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f34c469c-95a6-4134-b889-d20cbe59c1a3" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652644 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b6051c-d265-4cd2-b08b-b64054880109" containerName="maas-api" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652653 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa7be5f2-303b-41de-972c-4a36f84f127d" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652665 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="580aade9-3f41-41ff-a977-4a10a7814421" containerName="payload-processing" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652677 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652687 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" containerName="istio-proxy" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652696 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6862aa9e-2401-4f55-b63d-25435bf3eac4" containerName="payload-pre-processing" Oct 30 20:11:49.653680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.652705 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="229609ef-e005-41c3-807a-54fa69b874d9" containerName="payload-processing" Oct 30 20:11:49.656986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.656964 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.664110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.661992 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae2147\"/\"openshift-service-ca.crt\"" Oct 30 20:11:49.664110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.662135 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae2147\"/\"auth-test-model-ae2147-kserve-self-signed-certs\"" Oct 30 20:11:49.664110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.662233 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae2147\"/\"default-dockercfg-5vvcp\"" Oct 30 20:11:49.664110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.662308 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae2147\"/\"kube-root-ca.crt\"" Oct 30 20:11:49.665151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665114 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665295 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665336 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665429 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665516 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xfz\" (UniqueName: \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665574 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665547 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.665651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665638 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.665696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665657 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dr2v\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-kube-api-access-7dr2v\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.665696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665672 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.665765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665698 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.665765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.665712 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:49.666933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.666908 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:11:49.704286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.704246 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/ready\": dial tcp 10.133.0.76:8000: connect: connection refused" Oct 30 20:11:49.735537 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.735503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.766883 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767012 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767035 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767142 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767221 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xfz\" (UniqueName: \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767247 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.767723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.767597 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.768271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.768005 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.768337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.768297 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.773740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.773343 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.773740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.773515 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.783871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.783837 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xfz\" (UniqueName: \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\") pod \"auth-test-model-ae2147-kserve-6888d97957-8nxc6\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:49.893992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:49.893949 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.006516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.006165 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:11:50.011686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.011659 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 20:11:50.015915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.015891 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:11:50.050095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.050032 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:50.065700 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.065662 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d421fe44-7234-405c-a2fc-e21240a4f05d" path="/var/lib/kubelet/pods/d421fe44-7234-405c-a2fc-e21240a4f05d/volumes" Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.091985 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\" (UniqueName: \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105275 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105324 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105393 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105432 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\" (UniqueName: \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105480 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105568 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\" (UniqueName: \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105610 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105671 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105707 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105796 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105856 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\" (UniqueName: \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105894 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.105932 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.106197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.106014 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\") pod \"5c087e21-e692-4eb2-8347-baa42ddc21d3\" (UID: \"5c087e21-e692-4eb2-8347-baa42ddc21d3\") " Oct 30 20:11:50.108285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.106053 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\") pod \"30e3bfe8-a903-4119-b1a6-46417f98910f\" (UID: \"30e3bfe8-a903-4119-b1a6-46417f98910f\") " Oct 30 20:11:50.108285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.107227 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.108285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.107259 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\") pod \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\" (UID: \"4e877fbf-b484-4cfc-93a9-b2e813a364b2\") " Oct 30 20:11:50.108883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.097644 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "kube-api-access-nmp96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:50.109164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.108861 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.109293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.109275 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.109839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.109820 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.110483 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.112210 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.112442 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.112684 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.115519 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.115832 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.115813 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.116628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.116600 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:50.121500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.121476 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:50.121679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.121485 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "kube-api-access-m5z9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:50.125521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.124544 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm" pod "5c087e21-e692-4eb2-8347-baa42ddc21d3" (UID: "5c087e21-e692-4eb2-8347-baa42ddc21d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.125521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.124671 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "kube-api-access-n2zsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:50.125896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.125876 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:50.127143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.126161 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm" pod "4e877fbf-b484-4cfc-93a9-b2e813a364b2" (UID: "4e877fbf-b484-4cfc-93a9-b2e813a364b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.127143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.126185 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm" pod "30e3bfe8-a903-4119-b1a6-46417f98910f" (UID: "30e3bfe8-a903-4119-b1a6-46417f98910f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209789 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209825 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209841 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209854 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209869 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmp96\" (UniqueName: \"kubernetes.io/projected/4e877fbf-b484-4cfc-93a9-b2e813a364b2-kube-api-access-nmp96\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209884 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209898 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209913 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c087e21-e692-4eb2-8347-baa42ddc21d3-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209928 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30e3bfe8-a903-4119-b1a6-46417f98910f-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209943 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209958 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5z9b\" (UniqueName: \"kubernetes.io/projected/5c087e21-e692-4eb2-8347-baa42ddc21d3-kube-api-access-m5z9b\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209972 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.209989 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30e3bfe8-a903-4119-b1a6-46417f98910f-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.210002 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.210016 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.210030 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zsk\" (UniqueName: \"kubernetes.io/projected/30e3bfe8-a903-4119-b1a6-46417f98910f-kube-api-access-n2zsk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.210042 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c087e21-e692-4eb2-8347-baa42ddc21d3-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.211744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.210081 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e877fbf-b484-4cfc-93a9-b2e813a364b2-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:50.353037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.353002 5733 generic.go:356] "Generic (PLEG): container finished" podID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerID="940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c" exitCode=0 Oct 30 20:11:50.353528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.353045 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" Oct 30 20:11:50.353528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.353095 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" event={"ID":"4e877fbf-b484-4cfc-93a9-b2e813a364b2","Type":"ContainerDied","Data":"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c"} Oct 30 20:11:50.353528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.353150 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp" event={"ID":"4e877fbf-b484-4cfc-93a9-b2e813a364b2","Type":"ContainerDied","Data":"63f55dd8c84c08f4b13463ec4102929989b2a8ea54d75a55c083f12880d76003"} Oct 30 20:11:50.353528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.353173 5733 scope.go:122] "RemoveContainer" containerID="940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c" Oct 30 20:11:50.355554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.355237 5733 generic.go:356] "Generic (PLEG): container finished" podID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerID="10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9" exitCode=0 Oct 30 20:11:50.355554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.355295 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" Oct 30 20:11:50.355554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.355448 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" event={"ID":"5c087e21-e692-4eb2-8347-baa42ddc21d3","Type":"ContainerDied","Data":"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9"} Oct 30 20:11:50.355554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.355482 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" event={"ID":"5c087e21-e692-4eb2-8347-baa42ddc21d3","Type":"ContainerDied","Data":"8745f0fad1b36ab6a29d27ec423673a1476572015f49685fe839f2ecbd5cbc03"} Oct 30 20:11:50.357791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.357483 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" event={"ID":"8b1adab2-ae8a-4b80-9273-97fb0d04e5ac","Type":"ContainerDied","Data":"772f4f37fc184cc69c024aa73f3096742e9f6970168cb03f5063b4651912532c"} Oct 30 20:11:50.357791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.357494 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq" Oct 30 20:11:50.359285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.359236 5733 generic.go:356] "Generic (PLEG): container finished" podID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerID="5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0" exitCode=0 Oct 30 20:11:50.359285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.359263 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" event={"ID":"30e3bfe8-a903-4119-b1a6-46417f98910f","Type":"ContainerDied","Data":"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0"} Oct 30 20:11:50.359285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.359281 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" Oct 30 20:11:50.359498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.359298 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr" event={"ID":"30e3bfe8-a903-4119-b1a6-46417f98910f","Type":"ContainerDied","Data":"1c75248c4e6cc901caa967f5905afc2bd1066a44e21c9c4a94c8377877ddcd3a"} Oct 30 20:11:50.367872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.367735 5733 scope.go:122] "RemoveContainer" containerID="940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c" Oct 30 20:11:50.368178 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:50.368152 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c\": container with ID starting with 940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c not found: ID does not exist" containerID="940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c" Oct 30 20:11:50.368304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.368187 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c"} err="failed to get container status \"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c\": rpc error: code = NotFound desc = could not find container \"940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c\": container with ID starting with 940102a7a50ef96bc265c272add0c1a322fec4ccb35c22e2c95541ed17a3988c not found: ID does not exist" Oct 30 20:11:50.368304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.368204 5733 scope.go:122] "RemoveContainer" containerID="10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9" Oct 30 20:11:50.383432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.382892 5733 scope.go:122] "RemoveContainer" containerID="10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9" Oct 30 20:11:50.383432 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:50.383286 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9\": container with ID starting with 10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9 not found: ID does not exist" containerID="10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9" Oct 30 20:11:50.383432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.383315 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9"} err="failed to get container status \"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9\": rpc error: code = NotFound desc = could not find container \"10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9\": container with ID starting with 10458aaa8f5a7a9f79f85a39a9a748ebf20ab4b73b37d7ce2ebebf534cf686f9 not found: ID does not exist" Oct 30 20:11:50.383432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.383334 5733 scope.go:122] "RemoveContainer" containerID="f343c6aecce68c616c96736e712eda12461cff3ea6b0d1d37c859d06999ee229" Oct 30 20:11:50.384215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.384199 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:11:50.389487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.389465 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-55d2e0-openshift-default-5f4b4c8b79-8ngmq"] Oct 30 20:11:50.394339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.394297 5733 scope.go:122] "RemoveContainer" containerID="5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0" Oct 30 20:11:50.399328 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.399307 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 20:11:50.406764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.406743 5733 scope.go:122] "RemoveContainer" containerID="5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0" Oct 30 20:11:50.407093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.407044 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/premium-w2-d5e3e5-kserve-657bb55ccf-lv9hr"] Oct 30 20:11:50.407093 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:50.407074 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0\": container with ID starting with 5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0 not found: ID does not exist" containerID="5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0" Oct 30 20:11:50.407206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.407109 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0"} err="failed to get container status \"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0\": rpc error: code = NotFound desc = could not find container \"5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0\": container with ID starting with 5aede5b15c9e66b746667d7b9b37dac1a32fbe851916604d73e05c1f6386a9a0 not found: ID does not exist" Oct 30 20:11:50.418122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.418099 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 20:11:50.422748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.422724 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/distinct-w2-d5e3e5-kserve-c67646f86-96xsp"] Oct 30 20:11:50.429706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.429656 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:11:50.435436 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:11:50.435403 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa02c1ea_cceb_4e21_a092_8abf6fa058ad.slice/crio-48109a431058bc09ba6614507afea5af42b36bdec9be1da85c16e0f44cc7554e WatchSource:0}: Error finding container 48109a431058bc09ba6614507afea5af42b36bdec9be1da85c16e0f44cc7554e: Status 404 returned error can't find the container with id 48109a431058bc09ba6614507afea5af42b36bdec9be1da85c16e0f44cc7554e Oct 30 20:11:50.436260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.436236 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 20:11:50.444573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.443565 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m"] Oct 30 20:11:50.791443 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.791357 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.929989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:50.929944 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-d5e3e5/simulator-w2-d5e3e5-kserve-8578b46b7d-sgw6m" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:11:51.372932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:51.372894 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" event={"ID":"fa02c1ea-cceb-4e21-a092-8abf6fa058ad","Type":"ContainerStarted","Data":"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99"} Oct 30 20:11:51.373451 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:51.372945 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" event={"ID":"fa02c1ea-cceb-4e21-a092-8abf6fa058ad","Type":"ContainerStarted","Data":"48109a431058bc09ba6614507afea5af42b36bdec9be1da85c16e0f44cc7554e"} Oct 30 20:11:51.398038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:51.397989 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" podStartSLOduration=2.397975077 podStartE2EDuration="2.397975077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:11:51.393961622 +0000 UTC m=+2471.461797191" watchObservedRunningTime="2026-10-30 20:11:51.397975077 +0000 UTC m=+2471.465810665" Oct 30 20:11:51.849837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:51.849802 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:51.950207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:51.950153 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:52.019380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.017709 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:52.056610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.056572 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" path="/var/lib/kubelet/pods/30e3bfe8-a903-4119-b1a6-46417f98910f/volumes" Oct 30 20:11:52.057259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.057238 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" path="/var/lib/kubelet/pods/4e877fbf-b484-4cfc-93a9-b2e813a364b2/volumes" Oct 30 20:11:52.057842 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.057820 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" path="/var/lib/kubelet/pods/5c087e21-e692-4eb2-8347-baa42ddc21d3/volumes" Oct 30 20:11:52.058919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.058895 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" path="/var/lib/kubelet/pods/8b1adab2-ae8a-4b80-9273-97fb0d04e5ac/volumes" Oct 30 20:11:52.059356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.059339 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:11:52.170185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.170037 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:52.592511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:52.592426 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:53.094883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:53.094849 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:53.850256 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:53.850219 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:53.882912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:53.882882 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 20:11:54.000137 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:54.000107 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:54.157516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:54.157430 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:54.894695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:54.894658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:55.035335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:55.035291 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:55.787442 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:55.787313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:56.389436 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:56.389396 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:11:56.389917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:56.389661 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" containerID="cri-o://0374c4c11664ec8670fd204cdb316d169f4f1ec81fc7ba1ac3af039ebcb5feac" gracePeriod=30 Oct 30 20:11:56.482443 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:56.482401 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:56.703771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:56.701080 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:56.868246 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:56.868201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.032423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.032335 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:11:57.057172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.057141 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.427210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.427173 5733 generic.go:356] "Generic (PLEG): container finished" podID="47b19494-6b48-41e6-956a-713a34b0dc10" containerID="0374c4c11664ec8670fd204cdb316d169f4f1ec81fc7ba1ac3af039ebcb5feac" exitCode=0 Oct 30 20:11:57.427616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.427255 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerDied","Data":"0374c4c11664ec8670fd204cdb316d169f4f1ec81fc7ba1ac3af039ebcb5feac"} Oct 30 20:11:57.427616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.427322 5733 scope.go:122] "RemoveContainer" containerID="4123dfd6f71460bffa0a1db28a31cd8cd233d73b1106d1eefee5dbeead0e61df" Oct 30 20:11:57.482831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.482791 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.678396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.678353 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.844316 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.844286 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:11:57.940316 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.940241 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\" (UniqueName: \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\") pod \"47b19494-6b48-41e6-956a-713a34b0dc10\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " Oct 30 20:11:57.940533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.940409 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\") pod \"47b19494-6b48-41e6-956a-713a34b0dc10\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " Oct 30 20:11:57.940607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.940549 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\") pod \"47b19494-6b48-41e6-956a-713a34b0dc10\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " Oct 30 20:11:57.940678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.940653 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\") pod \"47b19494-6b48-41e6-956a-713a34b0dc10\" (UID: \"47b19494-6b48-41e6-956a-713a34b0dc10\") " Oct 30 20:11:57.941315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.940954 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca" pod "47b19494-6b48-41e6-956a-713a34b0dc10" (UID: "47b19494-6b48-41e6-956a-713a34b0dc10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:57.941474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.941449 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b19494-6b48-41e6-956a-713a34b0dc10-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:57.942877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.942851 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls" pod "47b19494-6b48-41e6-956a-713a34b0dc10" (UID: "47b19494-6b48-41e6-956a-713a34b0dc10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:57.942997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.942872 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs" pod "47b19494-6b48-41e6-956a-713a34b0dc10" (UID: "47b19494-6b48-41e6-956a-713a34b0dc10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:11:57.942997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:57.942970 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh" pod "47b19494-6b48-41e6-956a-713a34b0dc10" (UID: "47b19494-6b48-41e6-956a-713a34b0dc10"). InnerVolumeSpecName "kube-api-access-5mjgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:58.043365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.043337 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:58.043365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.043363 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjgh\" (UniqueName: \"kubernetes.io/projected/47b19494-6b48-41e6-956a-713a34b0dc10-kube-api-access-5mjgh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:58.043365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.043374 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b19494-6b48-41e6-956a-713a34b0dc10-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:58.420239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.420203 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:58.436871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.436840 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" Oct 30 20:11:58.437275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.436879 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x" event={"ID":"47b19494-6b48-41e6-956a-713a34b0dc10","Type":"ContainerDied","Data":"2127bbaa810ed65d14f21f0ffcad86ebd044191916e25dab483af07f076e321a"} Oct 30 20:11:58.437275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.436941 5733 scope.go:122] "RemoveContainer" containerID="0374c4c11664ec8670fd204cdb316d169f4f1ec81fc7ba1ac3af039ebcb5feac" Oct 30 20:11:58.463778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.463752 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:11:58.467963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.467938 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-4f244d-6dbb95b6d-gkg7x"] Oct 30 20:11:58.621704 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.621656 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:58.698287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.698186 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:11:58.884145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.884107 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" containerID="cri-o://739bdec92ca443c7e6f44c89b76d3dd4ddefeac50228c0adbb0aef7f3270785f" gracePeriod=25 Oct 30 20:11:58.909539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:58.909504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:59.078809 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:59.078719 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:59.378987 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:11:59.378947 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:11:59.511575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:11:59.511541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:00.062140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:00.062103 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" path="/var/lib/kubelet/pods/47b19494-6b48-41e6-956a-713a34b0dc10/volumes" Oct 30 20:12:00.062469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:00.062454 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:12:00.069175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:00.069151 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:12:00.255362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:00.255294 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:00.467841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:00.467812 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:12:01.712753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:01.712715 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:01.949756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:01.949719 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:01.955113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:01.955045 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:02.599185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.599150 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:12:02.608286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.608251 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.626989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.626953 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:12:02.793641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.793608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.966094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.965961 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" containerID="cri-o://c6233051842bab00187e40b121026c0d086b1ad1bdcf91427bb44fc1054ae406" gracePeriod=45 Oct 30 20:12:02.992281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:02.992189 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:03.419416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.419332 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:03.489639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.489597 5733 generic.go:356] "Generic (PLEG): container finished" podID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerID="c6233051842bab00187e40b121026c0d086b1ad1bdcf91427bb44fc1054ae406" exitCode=0 Oct 30 20:12:03.489920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.489898 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" event={"ID":"d550cab4-61b4-43c5-ae10-4eb9597cc7b0","Type":"ContainerDied","Data":"c6233051842bab00187e40b121026c0d086b1ad1bdcf91427bb44fc1054ae406"} Oct 30 20:12:03.861872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.861828 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.85:8000/ready\": dial tcp 10.133.0.85:8000: connect: connection refused" Oct 30 20:12:03.977728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.977694 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 20:12:03.977954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:03.977930 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" containerID="cri-o://47bfd6642f014f09aebab79a5592d5ed573a3601a186431685262be4093049e0" gracePeriod=30 Oct 30 20:12:04.081221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.081113 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.317602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.317556 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.500876 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.500842 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" event={"ID":"d550cab4-61b4-43c5-ae10-4eb9597cc7b0","Type":"ContainerDied","Data":"38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70"} Oct 30 20:12:04.501045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.500887 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38516751dec3068d3197c64ea2aaf0a094fb9a772bada4391fd8985458aefb70" Oct 30 20:12:04.502912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.502888 5733 generic.go:356] "Generic (PLEG): container finished" podID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerID="47bfd6642f014f09aebab79a5592d5ed573a3601a186431685262be4093049e0" exitCode=0 Oct 30 20:12:04.503088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.502962 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerDied","Data":"47bfd6642f014f09aebab79a5592d5ed573a3601a186431685262be4093049e0"} Oct 30 20:12:04.503088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.503028 5733 scope.go:122] "RemoveContainer" containerID="860fe8a77870582c820040e1db7560733db7ba2cfc6b3c83624c3c0f144b2307" Oct 30 20:12:04.554685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.554647 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.572036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.572011 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:12:04.633258 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633219 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633361 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633559 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633603 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633694 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:04.633828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633733 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\" (UniqueName: \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\") pod \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\" (UID: \"d550cab4-61b4-43c5-ae10-4eb9597cc7b0\") " Oct 30 20:12:04.633955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.633931 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:04.634368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.634342 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:04.634495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.634384 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:04.634710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.634684 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:04.636882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.636858 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:04.637014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.636910 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "kube-api-access-wmpjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:04.637229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.637204 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs" pod "d550cab4-61b4-43c5-ae10-4eb9597cc7b0" (UID: "d550cab4-61b4-43c5-ae10-4eb9597cc7b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:04.708396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.708264 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.727591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.727432 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.112:8443/health\": dial tcp 10.133.0.112:8443: connect: connection refused" Oct 30 20:12:04.735744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.735708 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpjp\" (UniqueName: \"kubernetes.io/projected/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-kube-api-access-wmpjp\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:04.735744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.735741 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:04.735976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.735755 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:04.735976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:04.735771 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d550cab4-61b4-43c5-ae10-4eb9597cc7b0-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:05.199634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.139491 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.438129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.438021 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:12:05.517498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.517455 5733 generic.go:356] "Generic (PLEG): container finished" podID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerID="1edfb42b38ed41db97908a3580b4546b999ed0c797f4ab555f90a0f9861ebf85" exitCode=0 Oct 30 20:12:05.517694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.517529 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" event={"ID":"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa","Type":"ContainerDied","Data":"1edfb42b38ed41db97908a3580b4546b999ed0c797f4ab555f90a0f9861ebf85"} Oct 30 20:12:05.519107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.519087 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" Oct 30 20:12:05.519107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.519093 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs" event={"ID":"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18","Type":"ContainerDied","Data":"280df8ceef2d4a5484429d18f55006123a475f859b749c82bd1843496ce65d60"} Oct 30 20:12:05.519304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.519146 5733 scope.go:122] "RemoveContainer" containerID="47bfd6642f014f09aebab79a5592d5ed573a3601a186431685262be4093049e0" Oct 30 20:12:05.520978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.520955 5733 generic.go:356] "Generic (PLEG): container finished" podID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerID="7811792985c4220199a6babde04642377e65c376b6378e9c11f93f518cffa9ab" exitCode=0 Oct 30 20:12:05.521098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.521028 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" event={"ID":"1fe5eb2e-3136-4690-815f-ead77c1a5384","Type":"ContainerDied","Data":"7811792985c4220199a6babde04642377e65c376b6378e9c11f93f518cffa9ab"} Oct 30 20:12:05.521212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.521198 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr" Oct 30 20:12:05.546311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.546280 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\" (UniqueName: \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\") pod \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " Oct 30 20:12:05.546478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.546327 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\") pod \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " Oct 30 20:12:05.546478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.546410 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\") pod \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " Oct 30 20:12:05.546478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.546457 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\") pod \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\" (UID: \"9bfc13ba-74ce-4e27-af6f-17b6c1a24c18\") " Oct 30 20:12:05.546777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.546748 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca" pod "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" (UID: "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:05.547285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.547260 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:05.548698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.548669 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs" pod "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" (UID: "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:05.548789 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.548696 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9" pod "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" (UID: "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18"). InnerVolumeSpecName "kube-api-access-x2pb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:05.549016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.548996 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls" pod "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" (UID: "9bfc13ba-74ce-4e27-af6f-17b6c1a24c18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:05.549211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.549191 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 20:12:05.557110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.557086 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a1d885/distinct-2-w1-a1d885-kserve-86947dfb9d-8gdgr"] Oct 30 20:12:05.584888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.584855 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.649022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.648986 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2pb9\" (UniqueName: \"kubernetes.io/projected/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-kube-api-access-x2pb9\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:05.649022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.649019 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:05.649241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.649031 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:05.845240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.845198 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 20:12:05.853271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.853245 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-a8de3b-dcc4bb4b5-7twfs"] Oct 30 20:12:05.855474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:05.855447 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.007377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.007342 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.053139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.053032 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" path="/var/lib/kubelet/pods/9bfc13ba-74ce-4e27-af6f-17b6c1a24c18/volumes" Oct 30 20:12:06.053739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.053716 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" path="/var/lib/kubelet/pods/d550cab4-61b4-43c5-ae10-4eb9597cc7b0/volumes" Oct 30 20:12:06.166412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.166377 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.230503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.230444 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:12:06.233033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.231004 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" containerID="cri-o://9b3a85d39566bc8ea1d012da80d0c2abe4f538741360b532cad3601336a55e87" gracePeriod=30 Oct 30 20:12:06.338575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.338480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.439108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.439040 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 20:12:06.462401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.462372 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\") pod \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " Oct 30 20:12:06.462705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.462683 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\" (UniqueName: \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\") pod \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\" (UID: \"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa\") " Oct 30 20:12:06.462836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.462686 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume" pod "3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" (UID: "3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:06.463170 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.463152 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:06.530659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.530634 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" Oct 30 20:12:06.530830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.530636 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc" event={"ID":"3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa","Type":"ContainerDied","Data":"a07dac7575211bac9bbda063789fddfca1c5969a4a5123114615821a21a89a4b"} Oct 30 20:12:06.530830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.530772 5733 scope.go:122] "RemoveContainer" containerID="1edfb42b38ed41db97908a3580b4546b999ed0c797f4ab555f90a0f9861ebf85" Oct 30 20:12:06.533225 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.533202 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" event={"ID":"1fe5eb2e-3136-4690-815f-ead77c1a5384","Type":"ContainerDied","Data":"efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691"} Oct 30 20:12:06.533343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.533246 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efbfd77edb96738822c33769f46a35483f89f2028144842e6c407f983d20e691" Oct 30 20:12:06.617216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.617177 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4" pod "3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" (UID: "3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa"). InnerVolumeSpecName "kube-api-access-5b6l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:06.666586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.666549 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6l4\" (UniqueName: \"kubernetes.io/projected/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa-kube-api-access-5b6l4\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:06.707639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.707610 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.772492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.772451 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 20:12:06.868510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.868439 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\" (UniqueName: \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\") pod \"1fe5eb2e-3136-4690-815f-ead77c1a5384\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " Oct 30 20:12:06.868510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.868499 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\") pod \"1fe5eb2e-3136-4690-815f-ead77c1a5384\" (UID: \"1fe5eb2e-3136-4690-815f-ead77c1a5384\") " Oct 30 20:12:06.883049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.883014 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 20:12:06.924838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.924801 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-d5e3e5-58c47bb6f9-d25pc"] Oct 30 20:12:06.956563 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.956522 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55" pod "1fe5eb2e-3136-4690-815f-ead77c1a5384" (UID: "1fe5eb2e-3136-4690-815f-ead77c1a5384"). InnerVolumeSpecName "kube-api-access-vrd55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:06.956726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.956562 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume" pod "1fe5eb2e-3136-4690-815f-ead77c1a5384" (UID: "1fe5eb2e-3136-4690-815f-ead77c1a5384"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:06.970295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.970265 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe5eb2e-3136-4690-815f-ead77c1a5384-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:06.970295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:06.970292 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrd55\" (UniqueName: \"kubernetes.io/projected/1fe5eb2e-3136-4690-815f-ead77c1a5384-kube-api-access-vrd55\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:07.553362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.551669 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg" Oct 30 20:12:07.642996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.587473 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 20:12:07.642996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.599556 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" containerID="cri-o://4dddfd0903c2e64409656e5498618384143b72a74c38171ae766e6b7da07c1ce" gracePeriod=25 Oct 30 20:12:07.642996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.602763 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-d5e3e5-5c5fb66764-fwdrg"] Oct 30 20:12:07.642996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.607922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:07.642996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.627297 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" containerID="cri-o://78f444055ce64fe1646d83493086c6f1422e2aba200d308410827f191a4324ff" gracePeriod=25 Oct 30 20:12:07.875796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.875763 5733 patch_prober.go:29] interesting pod/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:12:07.875948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:07.875831 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:12:08.052741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.052707 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" path="/var/lib/kubelet/pods/1fe5eb2e-3136-4690-815f-ead77c1a5384/volumes" Oct 30 20:12:08.053169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.053154 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" path="/var/lib/kubelet/pods/3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa/volumes" Oct 30 20:12:08.562267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.562236 5733 generic.go:356] "Generic (PLEG): container finished" podID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerID="4dddfd0903c2e64409656e5498618384143b72a74c38171ae766e6b7da07c1ce" exitCode=0 Oct 30 20:12:08.562674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.562310 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" event={"ID":"382161a6-f53d-4795-b7c4-005dd3e1ca94","Type":"ContainerDied","Data":"4dddfd0903c2e64409656e5498618384143b72a74c38171ae766e6b7da07c1ce"} Oct 30 20:12:08.563975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.563952 5733 generic.go:356] "Generic (PLEG): container finished" podID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerID="78f444055ce64fe1646d83493086c6f1422e2aba200d308410827f191a4324ff" exitCode=0 Oct 30 20:12:08.564112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.564019 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" event={"ID":"c9306a89-0fc2-44cd-9aa2-96f5ec782c02","Type":"ContainerDied","Data":"78f444055ce64fe1646d83493086c6f1422e2aba200d308410827f191a4324ff"} Oct 30 20:12:08.699336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.699242 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:08.711052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.711022 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:08.894841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:08.894813 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.051522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.051489 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.274852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.274820 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 20:12:09.275040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.275021 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" containerID="cri-o://9d764348a646ef3fb5c03204bcb5098f3c795e6202ed2f6b595e236738edb827" gracePeriod=30 Oct 30 20:12:09.431526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.424645 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.496579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.496545 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:12:09.574785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.574759 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" Oct 30 20:12:09.575213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.574793 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8" event={"ID":"382161a6-f53d-4795-b7c4-005dd3e1ca94","Type":"ContainerDied","Data":"8a331bea8150ae62ef394e67a2e3be43c3e98cbfd635147fe91a5778589a941d"} Oct 30 20:12:09.575213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.574886 5733 scope.go:122] "RemoveContainer" containerID="4dddfd0903c2e64409656e5498618384143b72a74c38171ae766e6b7da07c1ce" Oct 30 20:12:09.576417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.576395 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" event={"ID":"c9306a89-0fc2-44cd-9aa2-96f5ec782c02","Type":"ContainerDied","Data":"addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d"} Oct 30 20:12:09.576535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.576439 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="addbf00c7dc0aa2f11fe4c11511b8e11daa4459701c2ce81fe0a2096a37d7b7d" Oct 30 20:12:09.623214 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.623188 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:12:09.627343 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.627318 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\") pod \"382161a6-f53d-4795-b7c4-005dd3e1ca94\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " Oct 30 20:12:09.627464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.627446 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\" (UniqueName: \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\") pod \"382161a6-f53d-4795-b7c4-005dd3e1ca94\" (UID: \"382161a6-f53d-4795-b7c4-005dd3e1ca94\") " Oct 30 20:12:09.627676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.627656 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume" pod "382161a6-f53d-4795-b7c4-005dd3e1ca94" (UID: "382161a6-f53d-4795-b7c4-005dd3e1ca94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:09.628014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.627983 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382161a6-f53d-4795-b7c4-005dd3e1ca94-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:09.629483 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.629458 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh" pod "382161a6-f53d-4795-b7c4-005dd3e1ca94" (UID: "382161a6-f53d-4795-b7c4-005dd3e1ca94"). InnerVolumeSpecName "kube-api-access-8mffh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:09.728456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.728429 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\") pod \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " Oct 30 20:12:09.728689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.728673 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\" (UniqueName: \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\") pod \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\" (UID: \"c9306a89-0fc2-44cd-9aa2-96f5ec782c02\") " Oct 30 20:12:09.728822 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.728785 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume" pod "c9306a89-0fc2-44cd-9aa2-96f5ec782c02" (UID: "c9306a89-0fc2-44cd-9aa2-96f5ec782c02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:09.729236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.729216 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mffh\" (UniqueName: \"kubernetes.io/projected/382161a6-f53d-4795-b7c4-005dd3e1ca94-kube-api-access-8mffh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:09.729368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.729240 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:09.731229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.731203 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6" pod "c9306a89-0fc2-44cd-9aa2-96f5ec782c02" (UID: "c9306a89-0fc2-44cd-9aa2-96f5ec782c02"). InnerVolumeSpecName "kube-api-access-v82q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:09.830948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.830917 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v82q6\" (UniqueName: \"kubernetes.io/projected/c9306a89-0fc2-44cd-9aa2-96f5ec782c02-kube-api-access-v82q6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:09.904658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.904624 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:12:09.910073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:09.910036 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-82885f-55777b478d-khbl8"] Oct 30 20:12:10.043615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.043533 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.052547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.052519 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" path="/var/lib/kubelet/pods/382161a6-f53d-4795-b7c4-005dd3e1ca94/volumes" Oct 30 20:12:10.467413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.467377 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.585805 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.585773 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s" Oct 30 20:12:10.608741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.608709 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:12:10.616509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.616482 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-82885f-6d75b55b99-gr92s"] Oct 30 20:12:10.694618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.694580 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.760824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.760745 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:12:10.761146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.761108 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" containerID="cri-o://8ebf90152de528046ce879093a911d81cfb6a204ad85a6d76442badc967fe29d" gracePeriod=30 Oct 30 20:12:10.846662 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.846631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.987197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:10.987159 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.190435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.190358 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.402159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.402121 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.557069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.556979 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.595150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.595121 5733 generic.go:356] "Generic (PLEG): container finished" podID="e7520762-fb61-4512-875b-f376e9a2bbda" containerID="9b3a85d39566bc8ea1d012da80d0c2abe4f538741360b532cad3601336a55e87" exitCode=0 Oct 30 20:12:11.595510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.595166 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" event={"ID":"e7520762-fb61-4512-875b-f376e9a2bbda","Type":"ContainerDied","Data":"9b3a85d39566bc8ea1d012da80d0c2abe4f538741360b532cad3601336a55e87"} Oct 30 20:12:11.759621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.759591 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.955398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:11.955360 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:12.051209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.051178 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" path="/var/lib/kubelet/pods/c9306a89-0fc2-44cd-9aa2-96f5ec782c02/volumes" Oct 30 20:12:12.112809 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.112774 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.261746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.261658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.421267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.413523 5733 patch_prober.go:29] interesting pod/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:12:12.421267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.413673 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:12:12.486692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.486220 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.627838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.616832 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" event={"ID":"e7520762-fb61-4512-875b-f376e9a2bbda","Type":"ContainerDied","Data":"a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c"} Oct 30 20:12:12.627838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.616901 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1567095c939608bd49a4daad87058c38c54efbbce85d1a8e09470f18a7e6d6c" Oct 30 20:12:12.665582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.665546 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.781744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.781711 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:12:12.893695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893611 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.893695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893680 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.893918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893715 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.893918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893749 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.893918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893791 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.893918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893865 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.894152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.893987 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.894152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894025 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.894152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894042 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:12.894152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894077 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:12.894152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894108 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\") pod \"e7520762-fb61-4512-875b-f376e9a2bbda\" (UID: \"e7520762-fb61-4512-875b-f376e9a2bbda\") " Oct 30 20:12:12.894453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894378 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:12.894640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894578 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:12.894800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894769 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:12.894934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894916 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7520762-fb61-4512-875b-f376e9a2bbda-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.895014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894940 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.895014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894955 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.895014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894983 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.895014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.894998 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.896408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.896378 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "kube-api-access-6n6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:12.896516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.896405 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:12.896516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.896502 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:12.896632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.896575 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo" pod "e7520762-fb61-4512-875b-f376e9a2bbda" (UID: "e7520762-fb61-4512-875b-f376e9a2bbda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:12.996605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.996570 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.996605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.996601 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n6lv\" (UniqueName: \"kubernetes.io/projected/e7520762-fb61-4512-875b-f376e9a2bbda-kube-api-access-6n6lv\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.996605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.996612 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7520762-fb61-4512-875b-f376e9a2bbda-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.996827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:12.996621 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7520762-fb61-4512-875b-f376e9a2bbda-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:13.187778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:13.187694 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:13.328229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:13.328178 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:13.624703 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:13.624675 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6" Oct 30 20:12:13.649800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:13.649770 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:12:13.662177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:13.662152 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-4f244d-openshift-default-66b79d7575-dnww6"] Oct 30 20:12:14.050904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:14.050826 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" path="/var/lib/kubelet/pods/e7520762-fb61-4512-875b-f376e9a2bbda/volumes" Oct 30 20:12:14.151920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:14.151886 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:14.310341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:14.310265 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:14.551865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:14.551832 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.214263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.214230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.406878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.406849 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.650717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.650685 5733 generic.go:356] "Generic (PLEG): container finished" podID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerID="9d764348a646ef3fb5c03204bcb5098f3c795e6202ed2f6b595e236738edb827" exitCode=0 Oct 30 20:12:15.650885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.650763 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" event={"ID":"a74bb1f9-6fa8-4a48-92ef-62b50d195958","Type":"ContainerDied","Data":"9d764348a646ef3fb5c03204bcb5098f3c795e6202ed2f6b595e236738edb827"} Oct 30 20:12:15.650885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.650840 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" event={"ID":"a74bb1f9-6fa8-4a48-92ef-62b50d195958","Type":"ContainerDied","Data":"1e65fb432832c403d4164a966ba85ba67b78264adc608aec31202897216b1a22"} Oct 30 20:12:15.650885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.650860 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e65fb432832c403d4164a966ba85ba67b78264adc608aec31202897216b1a22" Oct 30 20:12:15.889076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.889032 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 20:12:15.935309 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935278 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935401 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935467 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935500 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935533 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935574 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935627 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.935844 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935790 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:15.935949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.935892 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:15.936151 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936129 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:15.936251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936179 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:15.936468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936445 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:15.936877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936853 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:15.936983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936883 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:15.936983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936900 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:15.936983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.936916 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:15.938087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.938042 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:15.938198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.938174 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:15.938257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.938199 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "kube-api-access-sq665". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:15.938344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:15.938326 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.038386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038356 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\") pod \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\" (UID: \"a74bb1f9-6fa8-4a48-92ef-62b50d195958\") " Oct 30 20:12:16.038707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038685 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert" pod "a74bb1f9-6fa8-4a48-92ef-62b50d195958" (UID: "a74bb1f9-6fa8-4a48-92ef-62b50d195958"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:16.038899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038884 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.038944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038901 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.038944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038913 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq665\" (UniqueName: \"kubernetes.io/projected/a74bb1f9-6fa8-4a48-92ef-62b50d195958-kube-api-access-sq665\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.038944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038924 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.038944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.038932 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74bb1f9-6fa8-4a48-92ef-62b50d195958-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.275590 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.275497 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.392702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.392668 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 20:12:16.406150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.406120 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 20:12:16.449315 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.449273 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.607102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.607050 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:12:16.607397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.607349 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" containerID="cri-o://73ead69171fb2a4a47423caf7e8e97fb720d3218ff07168624483c08c392cd5b" gracePeriod=30 Oct 30 20:12:16.616705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.616676 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.663275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.663244 5733 generic.go:356] "Generic (PLEG): container finished" podID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerID="8ebf90152de528046ce879093a911d81cfb6a204ad85a6d76442badc967fe29d" exitCode=0 Oct 30 20:12:16.663426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.663325 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerDied","Data":"8ebf90152de528046ce879093a911d81cfb6a204ad85a6d76442badc967fe29d"} Oct 30 20:12:16.663426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.663398 5733 scope.go:122] "RemoveContainer" containerID="5952e30aa9cfb6a17ce055ac00ba93d8c9904548c52d436e8214f25ec120003c" Oct 30 20:12:16.663546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.663498 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f" Oct 30 20:12:16.692664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.692628 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 20:12:16.701087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.700976 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-d5e3e5-openshift-default-6b46f4675f-kxb8f"] Oct 30 20:12:16.773079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.765601 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.874118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:16.874014 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:12:17.105624 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.105587 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.357685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.357658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.458406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.458317 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:12:17.458606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.458571 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" containerID="cri-o://5cb3f69e00b4e951db23cb6ee00f063211125d1ecba3bab270496754f56b9d69" gracePeriod=30 Oct 30 20:12:17.585091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.585026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.650709 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.650674 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:12:17.676275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.676241 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" event={"ID":"b78a1582-b7a4-4742-a25a-94209cae3c42","Type":"ContainerDied","Data":"d2988e71f2a7f997235f4863d576425f1bff04c26f6cee71c3fa6b765c8c9ea5"} Oct 30 20:12:17.676467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.676278 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm" Oct 30 20:12:17.676467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.676309 5733 scope.go:122] "RemoveContainer" containerID="8ebf90152de528046ce879093a911d81cfb6a204ad85a6d76442badc967fe29d" Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701139 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701266 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701361 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701405 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701488 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701522 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.701634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701600 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.702140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701658 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.702140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.701846 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\") pod \"b78a1582-b7a4-4742-a25a-94209cae3c42\" (UID: \"b78a1582-b7a4-4742-a25a-94209cae3c42\") " Oct 30 20:12:17.702934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.702903 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:17.703985 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.703956 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:17.704326 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.704302 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:17.704534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.704512 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:17.705236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.705152 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:17.709856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.707328 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "kube-api-access-t8hzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:17.709856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.707400 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:17.709856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.708543 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:17.714595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.714567 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo" pod "b78a1582-b7a4-4742-a25a-94209cae3c42" (UID: "b78a1582-b7a4-4742-a25a-94209cae3c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806132 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806171 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806191 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806206 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806220 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806234 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806247 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a1582-b7a4-4742-a25a-94209cae3c42-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806262 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8hzt\" (UniqueName: \"kubernetes.io/projected/b78a1582-b7a4-4742-a25a-94209cae3c42-kube-api-access-t8hzt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.806351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:17.806278 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a1582-b7a4-4742-a25a-94209cae3c42-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:18.011351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.011321 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:12:18.016379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.016354 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d7a733-openshift-default-6d6986c96c-dgbwm"] Oct 30 20:12:18.052485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.052454 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" path="/var/lib/kubelet/pods/a74bb1f9-6fa8-4a48-92ef-62b50d195958/volumes" Oct 30 20:12:18.053069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.053032 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" path="/var/lib/kubelet/pods/b78a1582-b7a4-4742-a25a-94209cae3c42/volumes" Oct 30 20:12:18.099444 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.099412 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.308721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.308618 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.688121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.688081 5733 generic.go:356] "Generic (PLEG): container finished" podID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerID="5cb3f69e00b4e951db23cb6ee00f063211125d1ecba3bab270496754f56b9d69" exitCode=0 Oct 30 20:12:18.688121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.688093 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerDied","Data":"5cb3f69e00b4e951db23cb6ee00f063211125d1ecba3bab270496754f56b9d69"} Oct 30 20:12:18.688572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.688182 5733 scope.go:122] "RemoveContainer" containerID="8b507b1ead19dcb840e44375ad1df9731ba7279cdd00459ed2136279012d3f7d" Oct 30 20:12:18.699959 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.699936 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.701217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.701186 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:18.701312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.701297 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:12:18.958104 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:18.958050 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:12:19.021627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.021595 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\") pod \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " Oct 30 20:12:19.021790 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.021662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\") pod \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " Oct 30 20:12:19.021881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.021867 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\" (UniqueName: \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\") pod \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " Oct 30 20:12:19.021953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.021942 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\") pod \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\" (UID: \"33c8e7e1-6494-49a8-b61c-bb3098dcccde\") " Oct 30 20:12:19.022036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.022014 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca" pod "33c8e7e1-6494-49a8-b61c-bb3098dcccde" (UID: "33c8e7e1-6494-49a8-b61c-bb3098dcccde"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:19.022534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.022508 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/33c8e7e1-6494-49a8-b61c-bb3098dcccde-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:19.024237 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.024205 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd" pod "33c8e7e1-6494-49a8-b61c-bb3098dcccde" (UID: "33c8e7e1-6494-49a8-b61c-bb3098dcccde"). InnerVolumeSpecName "kube-api-access-v5cgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:19.024697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.024675 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls" pod "33c8e7e1-6494-49a8-b61c-bb3098dcccde" (UID: "33c8e7e1-6494-49a8-b61c-bb3098dcccde"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:19.024787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.024746 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs" pod "33c8e7e1-6494-49a8-b61c-bb3098dcccde" (UID: "33c8e7e1-6494-49a8-b61c-bb3098dcccde"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:19.124022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.123987 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5cgd\" (UniqueName: \"kubernetes.io/projected/33c8e7e1-6494-49a8-b61c-bb3098dcccde-kube-api-access-v5cgd\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:19.124022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.124015 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:19.124022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.124025 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33c8e7e1-6494-49a8-b61c-bb3098dcccde-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:19.355841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.355812 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:19.602822 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.602791 5733 scope.go:122] "RemoveContainer" containerID="9b3a85d39566bc8ea1d012da80d0c2abe4f538741360b532cad3601336a55e87" Oct 30 20:12:19.696662 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.696577 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" Oct 30 20:12:19.696662 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.696595 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk" event={"ID":"33c8e7e1-6494-49a8-b61c-bb3098dcccde","Type":"ContainerDied","Data":"fcdf1350e6b3e6328fabe43827bcbeb557992a173c5b1122a483ce8900906fdc"} Oct 30 20:12:19.696662 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.696658 5733 scope.go:122] "RemoveContainer" containerID="5cb3f69e00b4e951db23cb6ee00f063211125d1ecba3bab270496754f56b9d69" Oct 30 20:12:19.698559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.698531 5733 generic.go:356] "Generic (PLEG): container finished" podID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerID="739bdec92ca443c7e6f44c89b76d3dd4ddefeac50228c0adbb0aef7f3270785f" exitCode=0 Oct 30 20:12:19.698668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.698571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" event={"ID":"7733aeff-8a16-47ab-ae96-b3ee56b1ef38","Type":"ContainerDied","Data":"739bdec92ca443c7e6f44c89b76d3dd4ddefeac50228c0adbb0aef7f3270785f"} Oct 30 20:12:19.742187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.742152 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:19.775857 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.775829 5733 scope.go:122] "RemoveContainer" containerID="c6233051842bab00187e40b121026c0d086b1ad1bdcf91427bb44fc1054ae406" Oct 30 20:12:19.782572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.782545 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:12:19.793434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.793412 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-77b4df-7c66c8cf45-xjjnk"] Oct 30 20:12:19.850219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.850189 5733 scope.go:122] "RemoveContainer" containerID="7811792985c4220199a6babde04642377e65c376b6378e9c11f93f518cffa9ab" Oct 30 20:12:19.937808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:19.937778 5733 scope.go:122] "RemoveContainer" containerID="9d764348a646ef3fb5c03204bcb5098f3c795e6202ed2f6b595e236738edb827" Oct 30 20:12:20.124895 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.124863 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" path="/var/lib/kubelet/pods/33c8e7e1-6494-49a8-b61c-bb3098dcccde/volumes" Oct 30 20:12:20.413418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.413390 5733 scope.go:122] "RemoveContainer" containerID="78f444055ce64fe1646d83493086c6f1422e2aba200d308410827f191a4324ff" Oct 30 20:12:20.674785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.601278 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 20:12:20.713535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.713509 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" Oct 30 20:12:20.713961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.713541 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc" event={"ID":"7733aeff-8a16-47ab-ae96-b3ee56b1ef38","Type":"ContainerDied","Data":"27ad818d1a5b6dca4ef29b9eb641d728aac405699af455ce07b211ec7370b660"} Oct 30 20:12:20.713961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.713606 5733 scope.go:122] "RemoveContainer" containerID="739bdec92ca443c7e6f44c89b76d3dd4ddefeac50228c0adbb0aef7f3270785f" Oct 30 20:12:20.746296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.746269 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\") pod \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " Oct 30 20:12:20.746504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.746435 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\" (UniqueName: \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\") pod \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\" (UID: \"7733aeff-8a16-47ab-ae96-b3ee56b1ef38\") " Oct 30 20:12:20.746625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.746604 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume" pod "7733aeff-8a16-47ab-ae96-b3ee56b1ef38" (UID: "7733aeff-8a16-47ab-ae96-b3ee56b1ef38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:20.746957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.746938 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:20.748558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.748538 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn" pod "7733aeff-8a16-47ab-ae96-b3ee56b1ef38" (UID: "7733aeff-8a16-47ab-ae96-b3ee56b1ef38"). InnerVolumeSpecName "kube-api-access-dvbmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:20.848089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.848032 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbmn\" (UniqueName: \"kubernetes.io/projected/7733aeff-8a16-47ab-ae96-b3ee56b1ef38-kube-api-access-dvbmn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:20.848434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:20.848413 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.100376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.055752 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 20:12:21.100376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.059324 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-a1d885-5586568475-mcvlc"] Oct 30 20:12:21.575434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.575396 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.686716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.686690 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 20:12:21.697400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.697365 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 20:12:21.775291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.775259 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.874907 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.874827 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" containerID="cri-o://cde33313a9ac503b0a84a13efd688ee7ef3121b45f8c55a793d9e31695da2c7c" gracePeriod=25 Oct 30 20:12:21.938756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.938688 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 20:12:21.938941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.938916 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" containerID="cri-o://397260cd646b5afb2919babcb13d1c99fbc636b9772d7a26649b696f99f20f7d" gracePeriod=30 Oct 30 20:12:21.945940 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:12:21.945922 5733 logging.go:55] [core] [Channel #6565 SubChannel #6566]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.164:50052", ServerName: "10.133.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.164:50052: connect: connection refused" Oct 30 20:12:21.971009 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:21.970981 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.055272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.055236 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" path="/var/lib/kubelet/pods/7733aeff-8a16-47ab-ae96-b3ee56b1ef38/volumes" Oct 30 20:12:22.147510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.147477 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.736965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.736933 5733 generic.go:356] "Generic (PLEG): container finished" podID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerID="cde33313a9ac503b0a84a13efd688ee7ef3121b45f8c55a793d9e31695da2c7c" exitCode=0 Oct 30 20:12:22.737173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.737016 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" event={"ID":"d05d05bb-294b-4d00-a60f-98fd0d84bfdb","Type":"ContainerDied","Data":"cde33313a9ac503b0a84a13efd688ee7ef3121b45f8c55a793d9e31695da2c7c"} Oct 30 20:12:22.738605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.738584 5733 generic.go:356] "Generic (PLEG): container finished" podID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerID="73ead69171fb2a4a47423caf7e8e97fb720d3218ff07168624483c08c392cd5b" exitCode=0 Oct 30 20:12:22.738739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.738661 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" event={"ID":"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9","Type":"ContainerDied","Data":"73ead69171fb2a4a47423caf7e8e97fb720d3218ff07168624483c08c392cd5b"} Oct 30 20:12:22.946801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:22.946754 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.164:50052\" within 1s: context deadline exceeded" Oct 30 20:12:22.947280 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:12:22.946860 5733 logging.go:55] [core] [Channel #6565 SubChannel #6566]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.164:50052", ServerName: "10.133.0.164:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.164:50052: operation was canceled" Oct 30 20:12:23.004867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.004838 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:23.587161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.587051 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:12:23.701817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.701780 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.701859 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.701931 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.701965 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702022 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702149 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702216 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702290 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702319 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:23.702390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702327 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\") pod \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\" (UID: \"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9\") " Oct 30 20:12:23.702390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702329 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:23.702390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702317 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:23.702629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702570 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:23.702685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702626 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:23.703013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.702992 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.703101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.703014 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.703101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.703024 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.703101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.703034 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.703101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.703043 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.704335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.704313 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:23.704900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.704882 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:23.704963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.704907 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:23.704963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.704918 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z" pod "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" (UID: "154a804d-f91a-4e31-bd7c-1bdc1e7f29e9"). InnerVolumeSpecName "kube-api-access-qzv5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:23.747107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.747079 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" event={"ID":"d05d05bb-294b-4d00-a60f-98fd0d84bfdb","Type":"ContainerDied","Data":"642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f"} Oct 30 20:12:23.747232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.747129 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="642591d6aee54889786290483a41468b52f229e95d26c2d72f46b1b2b9c8685f" Oct 30 20:12:23.748534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.748511 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" event={"ID":"154a804d-f91a-4e31-bd7c-1bdc1e7f29e9","Type":"ContainerDied","Data":"2ba1b189a90ec183d9e25100dc2f18746ae26bf246d8fcdbc31b28e49f76408a"} Oct 30 20:12:23.748534 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.748532 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd" Oct 30 20:12:23.748663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.748584 5733 scope.go:122] "RemoveContainer" containerID="73ead69171fb2a4a47423caf7e8e97fb720d3218ff07168624483c08c392cd5b" Oct 30 20:12:23.804021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.803993 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.804021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.804020 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzv5z\" (UniqueName: \"kubernetes.io/projected/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-kube-api-access-qzv5z\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.804195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.804030 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.804195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.804039 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:23.974133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:23.974103 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:12:24.032178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.032140 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:12:24.036631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.036602 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:24.037021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.036992 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-82885f-openshift-default-7478484b59-dcmsd"] Oct 30 20:12:24.053280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.053246 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" path="/var/lib/kubelet/pods/154a804d-f91a-4e31-bd7c-1bdc1e7f29e9/volumes" Oct 30 20:12:24.109051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.109021 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\") pod \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " Oct 30 20:12:24.109254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.109107 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\" (UniqueName: \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\") pod \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\" (UID: \"d05d05bb-294b-4d00-a60f-98fd0d84bfdb\") " Oct 30 20:12:24.109462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.109430 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume" pod "d05d05bb-294b-4d00-a60f-98fd0d84bfdb" (UID: "d05d05bb-294b-4d00-a60f-98fd0d84bfdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:24.109667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.109650 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:24.111236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.111211 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n" pod "d05d05bb-294b-4d00-a60f-98fd0d84bfdb" (UID: "d05d05bb-294b-4d00-a60f-98fd0d84bfdb"). InnerVolumeSpecName "kube-api-access-8rl2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:24.211830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.211750 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rl2n\" (UniqueName: \"kubernetes.io/projected/d05d05bb-294b-4d00-a60f-98fd0d84bfdb-kube-api-access-8rl2n\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:24.464017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.463931 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:24.756782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.756708 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz" Oct 30 20:12:24.792674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.792644 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:12:24.796554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:24.796517 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-77b4df-694f4cfff4-kkphz"] Oct 30 20:12:25.206980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:25.206936 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.378307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:25.378275 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.527801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:25.527723 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.050330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.050294 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" path="/var/lib/kubelet/pods/d05d05bb-294b-4d00-a60f-98fd0d84bfdb/volumes" Oct 30 20:12:26.278980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.278945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.440672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.440636 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.596215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.596185 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.687951 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.687913 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" containerID="cri-o://e9789732e4398411d3ef1eb358945c345562b92e11eb20b3d22ca454e1fa91ff" gracePeriod=25 Oct 30 20:12:26.698521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:26.698439 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" containerID="cri-o://45fd828105b297d51743901958169d9c03629a4daec77821757ed259fa55e6e0" gracePeriod=25 Oct 30 20:12:27.276392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:27.276353 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 20:12:27.473279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:27.473226 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.625465 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:27.625408 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.783097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:27.782975 5733 generic.go:356] "Generic (PLEG): container finished" podID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerID="397260cd646b5afb2919babcb13d1c99fbc636b9772d7a26649b696f99f20f7d" exitCode=0 Oct 30 20:12:27.783097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:27.783068 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" event={"ID":"bdd45e19-244b-44e1-87c0-d18e32f1de37","Type":"ContainerDied","Data":"397260cd646b5afb2919babcb13d1c99fbc636b9772d7a26649b696f99f20f7d"} Oct 30 20:12:28.353531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.353496 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.408046 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.407973 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 20:12:28.464603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464566 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.464774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464618 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.464774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464685 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.464774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464755 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.465026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464786 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.465026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464927 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.465026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.464985 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.465217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.465035 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.465217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.465090 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\") pod \"bdd45e19-244b-44e1-87c0-d18e32f1de37\" (UID: \"bdd45e19-244b-44e1-87c0-d18e32f1de37\") " Oct 30 20:12:28.504044 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.504020 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.552100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.552025 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:28.552356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.552325 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:28.552584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.552337 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:28.552725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.552643 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:28.552855 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.552661 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:28.554968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.554940 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:28.559754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.558908 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:28.561452 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.561419 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "kube-api-access-wb44z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567249 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567297 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567313 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567327 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567341 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb44z\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-kube-api-access-wb44z\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567373 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd45e19-244b-44e1-87c0-d18e32f1de37-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567388 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.567402 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.576936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.572524 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token" pod "bdd45e19-244b-44e1-87c0-d18e32f1de37" (UID: "bdd45e19-244b-44e1-87c0-d18e32f1de37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:28.649022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.648988 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.669082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.668995 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd45e19-244b-44e1-87c0-d18e32f1de37-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:28.790019 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.789980 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:12:28.790337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.790302 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" containerID="cri-o://6e82609386271287934d9c795b6892c0b19f762044228933d9eb67c06ed03fb4" gracePeriod=30 Oct 30 20:12:28.795906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.795881 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" Oct 30 20:12:28.796045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.795916 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h" event={"ID":"bdd45e19-244b-44e1-87c0-d18e32f1de37","Type":"ContainerDied","Data":"79a94be5d59d2475233d5210d0fe82c0a18bc9f8fb5e02145bd6241557cfc5a9"} Oct 30 20:12:28.796045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.795980 5733 scope.go:122] "RemoveContainer" containerID="397260cd646b5afb2919babcb13d1c99fbc636b9772d7a26649b696f99f20f7d" Oct 30 20:12:28.857189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.857147 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 20:12:28.861181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.861151 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a1d885-openshift-default-7ff56d66f-gl57h"] Oct 30 20:12:28.873169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:28.873138 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.066761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:29.066716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.379188 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:12:29.379151 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:12:29.563702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:29.563669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.743254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:29.743174 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:29.789577 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:29.789548 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.049978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:30.049902 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" path="/var/lib/kubelet/pods/bdd45e19-244b-44e1-87c0-d18e32f1de37/volumes" Oct 30 20:12:30.147695 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:30.147668 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.358841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.358804 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.415745 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.415705 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" containerID="cri-o://fbf88ad62aa4b42357dffdb6435bcd91149b05f865c57bb788f3b4b15f37aabc" gracePeriod=45 Oct 30 20:12:31.426307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.426270 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" containerID="cri-o://bdfa3d6a341557552c83656e814a421ff9e5c0179cc42cc26ac9de55cc54af46" gracePeriod=45 Oct 30 20:12:31.823999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.823958 5733 generic.go:356] "Generic (PLEG): container finished" podID="86e76d74-7775-4b7b-9120-885c993ad626" containerID="bdfa3d6a341557552c83656e814a421ff9e5c0179cc42cc26ac9de55cc54af46" exitCode=0 Oct 30 20:12:31.824206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.824034 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" event={"ID":"86e76d74-7775-4b7b-9120-885c993ad626","Type":"ContainerDied","Data":"bdfa3d6a341557552c83656e814a421ff9e5c0179cc42cc26ac9de55cc54af46"} Oct 30 20:12:31.826217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.826189 5733 generic.go:356] "Generic (PLEG): container finished" podID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerID="fbf88ad62aa4b42357dffdb6435bcd91149b05f865c57bb788f3b4b15f37aabc" exitCode=0 Oct 30 20:12:31.826336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.826266 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" event={"ID":"64f65e7a-479c-44cd-9f40-5aea55deb3e1","Type":"ContainerDied","Data":"fbf88ad62aa4b42357dffdb6435bcd91149b05f865c57bb788f3b4b15f37aabc"} Oct 30 20:12:31.887673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.887638 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:31.904454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.904378 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:31.951326 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:31.951294 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused" Oct 30 20:12:32.083629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.083595 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz"] Oct 30 20:12:32.084177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084158 5733 cpu_manager.go:414] "containerMap: removing container" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:12:32.084177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084175 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084187 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084192 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084201 5733 cpu_manager.go:414] "containerMap: removing container" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084206 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084214 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084218 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084226 5733 cpu_manager.go:414] "containerMap: removing container" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084232 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084241 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084246 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084256 5733 cpu_manager.go:414] "containerMap: removing container" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084261 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084269 5733 cpu_manager.go:414] "containerMap: removing container" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084273 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084282 5733 cpu_manager.go:414] "containerMap: removing container" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084286 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084293 5733 cpu_manager.go:414] "containerMap: removing container" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084298 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084306 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084311 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1adab2-ae8a-4b80-9273-97fb0d04e5ac" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084324 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084328 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084335 5733 cpu_manager.go:414] "containerMap: removing container" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084340 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084346 5733 cpu_manager.go:414] "containerMap: removing container" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084351 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084360 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084366 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:12:32.084355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084376 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084380 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084390 5733 cpu_manager.go:414] "containerMap: removing container" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084395 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084404 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084409 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084418 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084424 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084436 5733 cpu_manager.go:414] "containerMap: removing container" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084441 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084470 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084477 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084592 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9306a89-0fc2-44cd-9aa2-96f5ec782c02" containerName="payload-pre-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084608 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084619 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e877fbf-b484-4cfc-93a9-b2e813a364b2" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084629 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4c6696-0306-4c89-9f3d-81fbdfa0ecfa" containerName="payload-pre-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084645 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c087e21-e692-4eb2-8347-baa42ddc21d3" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084653 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d550cab4-61b4-43c5-ae10-4eb9597cc7b0" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084659 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e31201-64ff-4e20-999c-7e3be595e84a" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084667 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084673 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05d05bb-294b-4d00-a60f-98fd0d84bfdb" containerName="payload-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084679 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084684 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7520762-fb61-4512-875b-f376e9a2bbda" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084690 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="154a804d-f91a-4e31-bd7c-1bdc1e7f29e9" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084695 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084702 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084709 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfc13ba-74ce-4e27-af6f-17b6c1a24c18" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084719 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe5eb2e-3136-4690-815f-ead77c1a5384" containerName="payload-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084727 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74bb1f9-6fa8-4a48-92ef-62b50d195958" containerName="istio-proxy" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084733 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e3bfe8-a903-4119-b1a6-46417f98910f" containerName="main" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084742 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="382161a6-f53d-4795-b7c4-005dd3e1ca94" containerName="payload-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084748 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084755 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7733aeff-8a16-47ab-ae96-b3ee56b1ef38" containerName="payload-pre-processing" Oct 30 20:12:32.086566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.084760 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdd45e19-244b-44e1-87c0-d18e32f1de37" containerName="istio-proxy" Oct 30 20:12:32.206490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.206423 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz"] Oct 30 20:12:32.206633 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.206540 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.212385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.212360 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:12:32.212589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.212574 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:12:32.276733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.276697 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" containerID="cri-o://188314e6a9045e9871b99971f0b2390a072bf291a8efb63d925c149b0d620fc1" gracePeriod=25 Oct 30 20:12:32.314088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.314036 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": dial tcp 10.133.0.63:8000: connect: connection refused" Oct 30 20:12:32.314400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.314380 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqqrh\" (UniqueName: \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.314532 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.314513 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.314644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.314624 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.415607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.415578 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqqrh\" (UniqueName: \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.415944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.415690 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.415944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.415740 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.416543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.416521 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.418142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.418119 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.438395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.438371 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqqrh\" (UniqueName: \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\") pod \"maas-api-revoke-sub-b40ceac1-6b8tz\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.474724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.474666 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:32.609283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.609245 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:32.837332 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.837299 5733 generic.go:356] "Generic (PLEG): container finished" podID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerID="188314e6a9045e9871b99971f0b2390a072bf291a8efb63d925c149b0d620fc1" exitCode=0 Oct 30 20:12:32.837495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.837377 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" event={"ID":"50a17c23-b0b0-4bd2-a9d2-82afacf2546d","Type":"ContainerDied","Data":"188314e6a9045e9871b99971f0b2390a072bf291a8efb63d925c149b0d620fc1"} Oct 30 20:12:32.838895 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.838869 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" event={"ID":"86e76d74-7775-4b7b-9120-885c993ad626","Type":"ContainerDied","Data":"c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2"} Oct 30 20:12:32.839028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.838904 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4b75e5be3e3536a004523245dbcca98c3945eed645666da8294484dd22aaac2" Oct 30 20:12:32.840211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.840190 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" event={"ID":"64f65e7a-479c-44cd-9f40-5aea55deb3e1","Type":"ContainerDied","Data":"8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3"} Oct 30 20:12:32.840320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.840224 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8827e70564f86e22d54df2436aeb94f56c6800548e7d5d212b09a86b5180c8d3" Oct 30 20:12:32.982986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:32.982954 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz"] Oct 30 20:12:32.983677 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:12:32.983648 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7e7bd8_88c7_465c_be3d_fce254debc65.slice/crio-1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9 WatchSource:0}: Error finding container 1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9: Status 404 returned error can't find the container with id 1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9 Oct 30 20:12:33.112538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.112504 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:12:33.224591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224565 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224722 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224600 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224722 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224683 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\" (UniqueName: \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224753 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224815 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224853 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\" (UniqueName: \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\") pod \"86e76d74-7775-4b7b-9120-885c993ad626\" (UID: \"86e76d74-7775-4b7b-9120-885c993ad626\") " Oct 30 20:12:33.224879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224855 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.224991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.224940 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.225197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.225175 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.225489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.225426 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.225489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.225459 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.225489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.225475 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.227237 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.227206 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:33.227340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.227284 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.227429 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.227406 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh" pod "86e76d74-7775-4b7b-9120-885c993ad626" (UID: "86e76d74-7775-4b7b-9120-885c993ad626"). InnerVolumeSpecName "kube-api-access-vq6lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:33.284118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.284092 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.327829 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.327890 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.327946 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328017 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\" (UniqueName: \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328089 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328129 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\" (UniqueName: \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\") pod \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\" (UID: \"64f65e7a-479c-44cd-9f40-5aea55deb3e1\") " Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328661 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86e76d74-7775-4b7b-9120-885c993ad626-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328684 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86e76d74-7775-4b7b-9120-885c993ad626-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.329087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.328701 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6lh\" (UniqueName: \"kubernetes.io/projected/86e76d74-7775-4b7b-9120-885c993ad626-kube-api-access-vq6lh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.329815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.329229 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:33.329890 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.329812 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.330306 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.329992 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.397226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.397193 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:33.430127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.430096 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.430127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.430126 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.461535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.461405 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.461535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.461478 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "kube-api-access-f24t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:33.461535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.461494 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:33.461860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.461548 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs" pod "64f65e7a-479c-44cd-9f40-5aea55deb3e1" (UID: "64f65e7a-479c-44cd-9f40-5aea55deb3e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:33.531873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.531805 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.531873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.531833 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f24t6\" (UniqueName: \"kubernetes.io/projected/64f65e7a-479c-44cd-9f40-5aea55deb3e1-kube-api-access-f24t6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.531873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.531846 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64f65e7a-479c-44cd-9f40-5aea55deb3e1-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.531873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.531856 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64f65e7a-479c-44cd-9f40-5aea55deb3e1-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.641700 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.641663 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 20:12:33.733704 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.733660 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\") pod \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " Oct 30 20:12:33.733881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.733764 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\" (UniqueName: \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\") pod \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\" (UID: \"50a17c23-b0b0-4bd2-a9d2-82afacf2546d\") " Oct 30 20:12:33.734038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.734010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume" pod "50a17c23-b0b0-4bd2-a9d2-82afacf2546d" (UID: "50a17c23-b0b0-4bd2-a9d2-82afacf2546d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:33.734197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.734180 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.735949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.735926 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt" pod "50a17c23-b0b0-4bd2-a9d2-82afacf2546d" (UID: "50a17c23-b0b0-4bd2-a9d2-82afacf2546d"). InnerVolumeSpecName "kube-api-access-4ttbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:33.835386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.835316 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ttbt\" (UniqueName: \"kubernetes.io/projected/50a17c23-b0b0-4bd2-a9d2-82afacf2546d-kube-api-access-4ttbt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:33.849042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.849013 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" event={"ID":"50a17c23-b0b0-4bd2-a9d2-82afacf2546d","Type":"ContainerDied","Data":"8a96adcb46c35d6868efc93b5db223563fe7506c4734f2e5816cc74bc4dcbc3b"} Oct 30 20:12:33.849042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.849030 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw" Oct 30 20:12:33.849263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.849143 5733 scope.go:122] "RemoveContainer" containerID="188314e6a9045e9871b99971f0b2390a072bf291a8efb63d925c149b0d620fc1" Oct 30 20:12:33.850878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.850845 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" event={"ID":"5a7e7bd8-88c7-465c-be3d-fce254debc65","Type":"ContainerStarted","Data":"1b57eee408478ffa2b56518386e31254d5f84a1fe38fc3d9a06c95e8930e0fcc"} Oct 30 20:12:33.850979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.850875 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4" Oct 30 20:12:33.850979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.850914 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" event={"ID":"5a7e7bd8-88c7-465c-be3d-fce254debc65","Type":"ContainerStarted","Data":"1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9"} Oct 30 20:12:33.850979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.850859 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc" Oct 30 20:12:33.886507 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.886462 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" podStartSLOduration=1.88645006 podStartE2EDuration="1.88645006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:12:33.883273748 +0000 UTC m=+2513.951109325" watchObservedRunningTime="2026-10-30 20:12:33.88645006 +0000 UTC m=+2513.954285629" Oct 30 20:12:33.899545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.899520 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 20:12:33.907110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.907090 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/distinct-w4-3f57c5-kserve-7d69fd4768-9mjn4"] Oct 30 20:12:33.918514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.918490 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 20:12:33.922224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.922203 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3f57c5/premium-w4-3f57c5-kserve-c49958c8f-85rhc"] Oct 30 20:12:33.935013 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.934986 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 20:12:33.938282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:33.938263 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b0d13a-7867874db6-c8mzw"] Oct 30 20:12:34.049511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.049486 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" path="/var/lib/kubelet/pods/50a17c23-b0b0-4bd2-a9d2-82afacf2546d/volumes" Oct 30 20:12:34.123477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.123437 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" path="/var/lib/kubelet/pods/64f65e7a-479c-44cd-9f40-5aea55deb3e1/volumes" Oct 30 20:12:34.125072 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.125035 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e76d74-7775-4b7b-9120-885c993ad626" path="/var/lib/kubelet/pods/86e76d74-7775-4b7b-9120-885c993ad626/volumes" Oct 30 20:12:34.253867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.253841 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.420224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.420182 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 20:12:34.510501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.510468 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.708949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.708908 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.862339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.862304 5733 generic.go:356] "Generic (PLEG): container finished" podID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerID="6e82609386271287934d9c795b6892c0b19f762044228933d9eb67c06ed03fb4" exitCode=0 Oct 30 20:12:34.862515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.862338 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerDied","Data":"6e82609386271287934d9c795b6892c0b19f762044228933d9eb67c06ed03fb4"} Oct 30 20:12:34.862515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.862403 5733 scope.go:122] "RemoveContainer" containerID="06977bec715dad5676938c01cef6f28c9d4b1a9d746da24f7e03ad44cf9754a4" Oct 30 20:12:34.865899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.865871 5733 generic.go:356] "Generic (PLEG): container finished" podID="5a7e7bd8-88c7-465c-be3d-fce254debc65" containerID="1b57eee408478ffa2b56518386e31254d5f84a1fe38fc3d9a06c95e8930e0fcc" exitCode=0 Oct 30 20:12:34.866007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.865950 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" event={"ID":"5a7e7bd8-88c7-465c-be3d-fce254debc65","Type":"ContainerDied","Data":"1b57eee408478ffa2b56518386e31254d5f84a1fe38fc3d9a06c95e8930e0fcc"} Oct 30 20:12:34.931335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:34.931306 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.089257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.089228 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.312924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.312890 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:12:35.350516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.350480 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.351575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351549 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.351783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351598 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.351783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351676 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.351783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351731 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.351783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351773 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.352086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351911 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.352086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.351981 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.352086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352013 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:35.352086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352036 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.352086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352086 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\") pod \"affa7e70-81df-440e-a795-9e0b5ee3ae27\" (UID: \"affa7e70-81df-440e-a795-9e0b5ee3ae27\") " Oct 30 20:12:35.352381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352128 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:35.352381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352177 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:35.352381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352369 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:35.352631 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352600 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:35.352730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352676 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.352730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352699 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/affa7e70-81df-440e-a795-9e0b5ee3ae27-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.352730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352714 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.352904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.352732 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.354400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.354377 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:35.354706 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.354686 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "kube-api-access-r7gz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:35.354765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.354714 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:35.359988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.359959 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy" pod "affa7e70-81df-440e-a795-9e0b5ee3ae27" (UID: "affa7e70-81df-440e-a795-9e0b5ee3ae27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:35.453929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.453890 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.453929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.453923 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.454145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.453938 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7gz8\" (UniqueName: \"kubernetes.io/projected/affa7e70-81df-440e-a795-9e0b5ee3ae27-kube-api-access-r7gz8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.454145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.453952 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.454145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.453964 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/affa7e70-81df-440e-a795-9e0b5ee3ae27-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:35.875939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.875906 5733 generic.go:356] "Generic (PLEG): container finished" podID="38afb684-e999-4c10-9281-62f633431cc6" containerID="e91cda68bca3bac20fe06da4495966a7f7b1e4b1c44993f8a6c43a40a9825f63" exitCode=0 Oct 30 20:12:35.876375 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.875984 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" event={"ID":"38afb684-e999-4c10-9281-62f633431cc6","Type":"ContainerDied","Data":"e91cda68bca3bac20fe06da4495966a7f7b1e4b1c44993f8a6c43a40a9825f63"} Oct 30 20:12:35.877630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.877604 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" event={"ID":"affa7e70-81df-440e-a795-9e0b5ee3ae27","Type":"ContainerDied","Data":"be24f0b9d2dc4d5a8034c95087058948a6b0f5c90c69470de644dc9fc68da5b5"} Oct 30 20:12:35.877630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.877628 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq" Oct 30 20:12:35.877834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.877666 5733 scope.go:122] "RemoveContainer" containerID="6e82609386271287934d9c795b6892c0b19f762044228933d9eb67c06ed03fb4" Oct 30 20:12:35.930652 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.930615 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:12:35.942800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.942768 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.942963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:35.942804 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-77b4df-openshift-default-7c844967cc-f2zrq"] Oct 30 20:12:36.051118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.051088 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" path="/var/lib/kubelet/pods/affa7e70-81df-440e-a795-9e0b5ee3ae27/volumes" Oct 30 20:12:36.286266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.286233 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:36.364081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.364031 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\") pod \"5a7e7bd8-88c7-465c-be3d-fce254debc65\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " Oct 30 20:12:36.364252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.364122 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\" (UniqueName: \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\") pod \"5a7e7bd8-88c7-465c-be3d-fce254debc65\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " Oct 30 20:12:36.364252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.364168 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\") pod \"5a7e7bd8-88c7-465c-be3d-fce254debc65\" (UID: \"5a7e7bd8-88c7-465c-be3d-fce254debc65\") " Oct 30 20:12:36.364456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.364427 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca" pod "5a7e7bd8-88c7-465c-be3d-fce254debc65" (UID: "5a7e7bd8-88c7-465c-be3d-fce254debc65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:36.364722 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.364699 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:36.366345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.366323 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca" pod "5a7e7bd8-88c7-465c-be3d-fce254debc65" (UID: "5a7e7bd8-88c7-465c-be3d-fce254debc65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:36.366473 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.366453 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh" pod "5a7e7bd8-88c7-465c-be3d-fce254debc65" (UID: "5a7e7bd8-88c7-465c-be3d-fce254debc65"). InnerVolumeSpecName "kube-api-access-qqqrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:36.465904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.465879 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqqrh\" (UniqueName: \"kubernetes.io/projected/5a7e7bd8-88c7-465c-be3d-fce254debc65-kube-api-access-qqqrh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:36.465904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.465905 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a7e7bd8-88c7-465c-be3d-fce254debc65-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:36.691339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.691259 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:36.832710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.832678 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:36.886768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.886743 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" Oct 30 20:12:36.887168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.886767 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz" event={"ID":"5a7e7bd8-88c7-465c-be3d-fce254debc65","Type":"ContainerDied","Data":"1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9"} Oct 30 20:12:36.887168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:36.886814 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d66363d162411c01901e3bf1569ef791b5bfcb1ec4f4ab47efed830815353d9" Oct 30 20:12:37.371165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.371129 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:37.512119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.512081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:37.743247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.743217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:37.896754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.896708 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" event={"ID":"38afb684-e999-4c10-9281-62f633431cc6","Type":"ContainerStarted","Data":"43f47c10624a4605442cf1f2a58003fc5ee3a92d758a42f08c6805f01bcd0a1c"} Oct 30 20:12:37.897260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.896805 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 20:12:37.909327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:37.909296 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:38.373942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:38.373910 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.054611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.054572 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.676796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.676765 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 20:12:39.689336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.689305 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 20:12:39.735358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.735299 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.743254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.743216 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:39.906325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:39.906285 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.209177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:40.209146 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.897268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:40.897229 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:41.359380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:41.359347 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 20:12:41.805402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:41.805305 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:41.887866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:41.887827 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:41.904527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:41.904497 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:42.802374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:42.802329 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:43.220037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:43.220000 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 20:12:43.245992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:43.245959 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 20:12:43.410381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:43.410342 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:43.551419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:43.551346 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:43.844508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:43.844478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:44.414197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:44.414166 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:44.677338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:44.677249 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" containerID="cri-o://140d65a139a52a8db1def49aa9fa17d8e3b0ae77326625834c9057e339e3174e" gracePeriod=25 Oct 30 20:12:44.689871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:44.689838 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" containerID="cri-o://a1306238a7d548e81974eab2be415bc43ed2aa12c82ffbb815fde40b039e1ff6" gracePeriod=25 Oct 30 20:12:44.918530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:44.918478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.055710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:45.055634 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.715510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:45.715440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:46.900469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:46.900415 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:46.968910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:46.968824 5733 generic.go:356] "Generic (PLEG): container finished" podID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerID="45fd828105b297d51743901958169d9c03629a4daec77821757ed259fa55e6e0" exitCode=0 Oct 30 20:12:46.969099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:46.968889 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" event={"ID":"b700fa9e-7b5a-42f7-9d54-551c147eccc8","Type":"ContainerDied","Data":"45fd828105b297d51743901958169d9c03629a4daec77821757ed259fa55e6e0"} Oct 30 20:12:46.970775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:46.970748 5733 generic.go:356] "Generic (PLEG): container finished" podID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerID="e9789732e4398411d3ef1eb358945c345562b92e11eb20b3d22ca454e1fa91ff" exitCode=0 Oct 30 20:12:46.970947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:46.970815 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" event={"ID":"bd91d8c7-be08-4975-b46b-70f110ca1bfc","Type":"ContainerDied","Data":"e9789732e4398411d3ef1eb358945c345562b92e11eb20b3d22ca454e1fa91ff"} Oct 30 20:12:47.966966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:47.966919 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:47.980251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:47.980215 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" event={"ID":"bd91d8c7-be08-4975-b46b-70f110ca1bfc","Type":"ContainerDied","Data":"4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f"} Oct 30 20:12:47.980419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:47.980269 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4092bb7d6927c36628081f0003604c73f246b994d09e1201c074cd77d706001f" Oct 30 20:12:47.981813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:47.981782 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" event={"ID":"b700fa9e-7b5a-42f7-9d54-551c147eccc8","Type":"ContainerDied","Data":"9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7"} Oct 30 20:12:47.981942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:47.981826 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9231a9ea8f1e57e9251266b3a6eb7e1d1c87c328d6d5dfc30689dc3dda4577f7" Oct 30 20:12:48.147346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.147313 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 20:12:48.308404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.308368 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\") pod \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " Oct 30 20:12:48.308587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.308554 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\" (UniqueName: \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\") pod \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\" (UID: \"bd91d8c7-be08-4975-b46b-70f110ca1bfc\") " Oct 30 20:12:48.308738 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.308714 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume" pod "bd91d8c7-be08-4975-b46b-70f110ca1bfc" (UID: "bd91d8c7-be08-4975-b46b-70f110ca1bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:48.308980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.308964 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd91d8c7-be08-4975-b46b-70f110ca1bfc-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:48.310701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.310678 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8" pod "bd91d8c7-be08-4975-b46b-70f110ca1bfc" (UID: "bd91d8c7-be08-4975-b46b-70f110ca1bfc"). InnerVolumeSpecName "kube-api-access-xw9l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:48.332268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.332244 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 20:12:48.410478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.410389 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw9l8\" (UniqueName: \"kubernetes.io/projected/bd91d8c7-be08-4975-b46b-70f110ca1bfc-kube-api-access-xw9l8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:48.511508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.511477 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\") pod \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " Oct 30 20:12:48.511709 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.511518 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\" (UniqueName: \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\") pod \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\" (UID: \"b700fa9e-7b5a-42f7-9d54-551c147eccc8\") " Oct 30 20:12:48.511996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.511969 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume" pod "b700fa9e-7b5a-42f7-9d54-551c147eccc8" (UID: "b700fa9e-7b5a-42f7-9d54-551c147eccc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:48.512183 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.512153 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b700fa9e-7b5a-42f7-9d54-551c147eccc8-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:48.514768 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.514737 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s" pod "b700fa9e-7b5a-42f7-9d54-551c147eccc8" (UID: "b700fa9e-7b5a-42f7-9d54-551c147eccc8"). InnerVolumeSpecName "kube-api-access-4vt6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:48.611897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.611851 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:48.613247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.613222 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vt6s\" (UniqueName: \"kubernetes.io/projected/b700fa9e-7b5a-42f7-9d54-551c147eccc8-kube-api-access-4vt6s\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:48.759482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.759393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:48.990019 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.989989 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr" Oct 30 20:12:48.990499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:48.989989 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt" Oct 30 20:12:49.010238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.010164 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:49.024747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.024692 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 20:12:49.028864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.028837 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3f57c5-6d7689c84-8zxvt"] Oct 30 20:12:49.038756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.038723 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 20:12:49.044626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.044598 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3f57c5-657f97f78c-4pblr"] Oct 30 20:12:49.441498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.441456 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" containerID="cri-o://ab194f18663215340d74d8ca2e3e3a0acbce4171615837b6a63db0b571421623" gracePeriod=45 Oct 30 20:12:49.743234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.743199 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:49.861167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.861130 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/ready\": dial tcp 10.133.0.77:8000: connect: connection refused" Oct 30 20:12:49.865707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.865672 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:49.998841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.998757 5733 generic.go:356] "Generic (PLEG): container finished" podID="d142cf77-146e-4719-8e79-001e99aead3c" containerID="ab194f18663215340d74d8ca2e3e3a0acbce4171615837b6a63db0b571421623" exitCode=0 Oct 30 20:12:49.999243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:49.998832 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" event={"ID":"d142cf77-146e-4719-8e79-001e99aead3c","Type":"ContainerDied","Data":"ab194f18663215340d74d8ca2e3e3a0acbce4171615837b6a63db0b571421623"} Oct 30 20:12:50.053116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:50.053077 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" path="/var/lib/kubelet/pods/b700fa9e-7b5a-42f7-9d54-551c147eccc8/volumes" Oct 30 20:12:50.053726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:50.053701 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" path="/var/lib/kubelet/pods/bd91d8c7-be08-4975-b46b-70f110ca1bfc/volumes" Oct 30 20:12:50.164146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:50.164111 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:50.901351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:50.901249 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:12:50.939886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:50.939816 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.008147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.008113 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" event={"ID":"d142cf77-146e-4719-8e79-001e99aead3c","Type":"ContainerDied","Data":"ed1ac0a33338e4eaae0454dfcacc680cc46005dceba36501c51b79ab2efe304c"} Oct 30 20:12:51.008548 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.008156 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh" Oct 30 20:12:51.008548 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.008173 5733 scope.go:122] "RemoveContainer" containerID="ab194f18663215340d74d8ca2e3e3a0acbce4171615837b6a63db0b571421623" Oct 30 20:12:51.040821 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.040792 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.040981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.040895 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.041132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041105 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.041291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041153 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:51.041291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041157 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.041291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041262 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.041505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041330 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\" (UniqueName: \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\") pod \"d142cf77-146e-4719-8e79-001e99aead3c\" (UID: \"d142cf77-146e-4719-8e79-001e99aead3c\") " Oct 30 20:12:51.041614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.041591 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:51.043087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.043032 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:51.043929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.043909 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.044011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.043939 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.045720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.045697 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:51.046040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.046008 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "kube-api-access-6rn58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:51.046407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.046387 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs" pod "d142cf77-146e-4719-8e79-001e99aead3c" (UID: "d142cf77-146e-4719-8e79-001e99aead3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:51.145317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.145283 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.145317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.145313 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d142cf77-146e-4719-8e79-001e99aead3c-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.145317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.145323 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d142cf77-146e-4719-8e79-001e99aead3c-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.145558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.145333 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rn58\" (UniqueName: \"kubernetes.io/projected/d142cf77-146e-4719-8e79-001e99aead3c-kube-api-access-6rn58\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:51.343761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.343733 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 20:12:51.354870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.354844 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-278ffa/distinct-2-w5-278ffa-kserve-c69ccf475-jppkh"] Oct 30 20:12:51.938787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.938742 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.952640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:51.952599 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:12:52.218542 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:52.218459 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d142cf77-146e-4719-8e79-001e99aead3c" path="/var/lib/kubelet/pods/d142cf77-146e-4719-8e79-001e99aead3c/volumes" Oct 30 20:12:52.691377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:52.691336 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.833039 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:52.833006 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.042991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.042899 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.270337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.270304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.328299 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.328218 5733 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:53.328462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.328304 5733 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:12:53.328808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.328789 5733 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6"} pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" containerMessage="Container manager failed liveness probe, will be restarted" Oct 30 20:12:53.328874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.328838 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" containerID="cri-o://666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6" gracePeriod=10 Oct 30 20:12:53.459038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.458998 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.850455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:53.850418 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:54.001793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:54.001764 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:54.370249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:54.370210 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:12:54.873834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:54.873798 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:55.826447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:55.826415 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.314160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.314118 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.555496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.555463 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 20:12:56.555698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.555672 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" containerID="cri-o://3d859bfbec7b6698c0d1a91bc28867785e1cc0eb7b910ba21f19f31cad71d644" gracePeriod=30 Oct 30 20:12:56.617672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.617638 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.772170 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.772135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.919642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:56.919555 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.061406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.061375 5733 generic.go:356] "Generic (PLEG): container finished" podID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerID="3d859bfbec7b6698c0d1a91bc28867785e1cc0eb7b910ba21f19f31cad71d644" exitCode=0 Oct 30 20:12:57.061565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.061455 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" event={"ID":"ddb50a54-7c85-49dc-abfd-d9234707efd1","Type":"ContainerDied","Data":"3d859bfbec7b6698c0d1a91bc28867785e1cc0eb7b910ba21f19f31cad71d644"} Oct 30 20:12:57.126230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.126184 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.291149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.291026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.577578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.577498 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.783124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.758673 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.950524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:57.950493 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 20:12:58.027936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.027906 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\") pod \"ddb50a54-7c85-49dc-abfd-d9234707efd1\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " Oct 30 20:12:58.028139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.027954 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\" (UniqueName: \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\") pod \"ddb50a54-7c85-49dc-abfd-d9234707efd1\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " Oct 30 20:12:58.028139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.028004 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\") pod \"ddb50a54-7c85-49dc-abfd-d9234707efd1\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " Oct 30 20:12:58.028139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.028028 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\") pod \"ddb50a54-7c85-49dc-abfd-d9234707efd1\" (UID: \"ddb50a54-7c85-49dc-abfd-d9234707efd1\") " Oct 30 20:12:58.028474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.028436 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca" pod "ddb50a54-7c85-49dc-abfd-d9234707efd1" (UID: "ddb50a54-7c85-49dc-abfd-d9234707efd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:58.030193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.030170 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls" pod "ddb50a54-7c85-49dc-abfd-d9234707efd1" (UID: "ddb50a54-7c85-49dc-abfd-d9234707efd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:58.030287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.030197 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs" pod "ddb50a54-7c85-49dc-abfd-d9234707efd1" (UID: "ddb50a54-7c85-49dc-abfd-d9234707efd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:58.030287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.030254 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr" pod "ddb50a54-7c85-49dc-abfd-d9234707efd1" (UID: "ddb50a54-7c85-49dc-abfd-d9234707efd1"). InnerVolumeSpecName "kube-api-access-8tvdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:58.069895 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.069872 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" Oct 30 20:12:58.070014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.069905 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb" event={"ID":"ddb50a54-7c85-49dc-abfd-d9234707efd1","Type":"ContainerDied","Data":"7afda34ed5c50399ffff3001f21dabc7fc2132ae8818c00c4e6bf04f4e2f42a3"} Oct 30 20:12:58.070014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.069960 5733 scope.go:122] "RemoveContainer" containerID="3d859bfbec7b6698c0d1a91bc28867785e1cc0eb7b910ba21f19f31cad71d644" Oct 30 20:12:58.093724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.093693 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 20:12:58.097008 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.096986 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-278ffa-69d44fb748-mnzpb"] Oct 30 20:12:58.129508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.129483 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:58.129508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.129507 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tvdr\" (UniqueName: \"kubernetes.io/projected/ddb50a54-7c85-49dc-abfd-d9234707efd1-kube-api-access-8tvdr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:58.129683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.129518 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddb50a54-7c85-49dc-abfd-d9234707efd1-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:58.129683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.129527 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddb50a54-7c85-49dc-abfd-d9234707efd1-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:58.241312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.241273 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" containerID="cri-o://857f0b4926b80f58c968766f1dbe7be8a3aa87d24ebba0da98d85dd47ae72141" gracePeriod=45 Oct 30 20:12:58.266912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.266859 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" containerID="cri-o://88d28d0c138dc51caa10e6e00fb5090c2eccb2357b72774451571b2f3677e47c" gracePeriod=45 Oct 30 20:12:58.394551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.394517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:58.552469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.552390 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:58.755553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.755516 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": dial tcp 10.133.0.79:8000: connect: connection refused" Oct 30 20:12:58.909880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:58.909843 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused" Oct 30 20:12:59.119216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.119171 5733 generic.go:356] "Generic (PLEG): container finished" podID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerID="857f0b4926b80f58c968766f1dbe7be8a3aa87d24ebba0da98d85dd47ae72141" exitCode=0 Oct 30 20:12:59.119912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.119844 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" event={"ID":"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa","Type":"ContainerDied","Data":"857f0b4926b80f58c968766f1dbe7be8a3aa87d24ebba0da98d85dd47ae72141"} Oct 30 20:12:59.122154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.122106 5733 generic.go:356] "Generic (PLEG): container finished" podID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerID="88d28d0c138dc51caa10e6e00fb5090c2eccb2357b72774451571b2f3677e47c" exitCode=0 Oct 30 20:12:59.122446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.122392 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" event={"ID":"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8","Type":"ContainerDied","Data":"88d28d0c138dc51caa10e6e00fb5090c2eccb2357b72774451571b2f3677e47c"} Oct 30 20:12:59.377730 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:12:59.377693 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:12:59.782708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.782632 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.837664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.837621 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 20:12:59.847478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847413 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.847478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847473 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.847760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847603 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.847760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847667 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\" (UniqueName: \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.847760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847708 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.847942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847756 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:59.847942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847770 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:59.847942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.847786 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\") pod \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\" (UID: \"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa\") " Oct 30 20:12:59.848111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.848006 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:59.848276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.848258 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:59.848365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.848302 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:59.848365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.848317 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:59.849935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.849910 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "kube-api-access-594d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:59.850036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.850017 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:59.850345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.850326 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs" pod "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" (UID: "d1c5d55c-b95f-4c74-9bbb-4d660b088cfa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:59.949602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.949567 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-594d9\" (UniqueName: \"kubernetes.io/projected/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-kube-api-access-594d9\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:59.949602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.949603 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:59.949850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:12:59.949618 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.049759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.049684 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" path="/var/lib/kubelet/pods/ddb50a54-7c85-49dc-abfd-d9234707efd1/volumes" Oct 30 20:13:00.140254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.140220 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" event={"ID":"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8","Type":"ContainerDied","Data":"4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5"} Oct 30 20:13:00.140679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.140266 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cca0a270b317e9f7fdd387fbb378e27747d9118c41a913acef6cdecf5da22e5" Oct 30 20:13:00.141681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.141654 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" event={"ID":"d1c5d55c-b95f-4c74-9bbb-4d660b088cfa","Type":"ContainerDied","Data":"e628c22770f21328ed59a671ca928aadfc1ed4acb6c412fca70e069cce2e6723"} Oct 30 20:13:00.141800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.141684 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf" Oct 30 20:13:00.141800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.141700 5733 scope.go:122] "RemoveContainer" containerID="857f0b4926b80f58c968766f1dbe7be8a3aa87d24ebba0da98d85dd47ae72141" Oct 30 20:13:00.164025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.163999 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:13:00.458406 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.458368 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:13:00.497511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.497476 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 20:13:00.506874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.506834 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/distinct-w6-5f933e-kserve-594b74cb-9fzpf"] Oct 30 20:13:00.556538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556509 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556566 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556617 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556715 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556774 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556806 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\" (UniqueName: \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\") pod \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\" (UID: \"5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8\") " Oct 30 20:13:00.556999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.556925 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:00.557303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.557006 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:00.557419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.557316 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:00.557486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.557461 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.557486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.557481 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.557586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.557496 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.558767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.558738 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:00.558874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.558846 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:00.559309 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.559292 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l" pod "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" (UID: "5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8"). InnerVolumeSpecName "kube-api-access-4zk5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:00.658811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.658776 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.658811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.658809 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.659033 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.658820 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zk5l\" (UniqueName: \"kubernetes.io/projected/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8-kube-api-access-4zk5l\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:00.693412 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.693362 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.910159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.910116 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:13:00.910339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:00.910202 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:13:01.149974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.149904 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq" Oct 30 20:13:01.175377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.175298 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 20:13:01.178415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.178393 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-5f933e/simulator-w6-5f933e-kserve-86bf55fd5b-pncbq"] Oct 30 20:13:01.363875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.363842 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 20:13:01.367669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.367646 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bd8695446-nbt67" Oct 30 20:13:01.721879 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.721840 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:01.950136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:01.950094 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:13:02.049693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:02.049599 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" path="/var/lib/kubelet/pods/5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8/volumes" Oct 30 20:13:02.050198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:02.050176 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" path="/var/lib/kubelet/pods/d1c5d55c-b95f-4c74-9bbb-4d660b088cfa/volumes" Oct 30 20:13:02.740103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:02.740051 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:02.968395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:02.968337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:03.191236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:03.191195 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:03.474502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:03.474390 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": read tcp 10.133.0.2:51792->10.133.0.35:8081: read: connection reset by peer" Oct 30 20:13:03.857937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:03.857903 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:04.173830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.173747 5733 generic.go:356] "Generic (PLEG): container finished" podID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerID="666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6" exitCode=137 Oct 30 20:13:04.173830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.173821 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerDied","Data":"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6"} Oct 30 20:13:04.174091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.173877 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerStarted","Data":"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e"} Oct 30 20:13:04.174091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.173900 5733 scope.go:122] "RemoveContainer" containerID="ebcc8bd4637a74a5fd5a68a80e8a393e1a543b7be28dd22b3bb8a4c3c41a99b1" Oct 30 20:13:04.174191 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.174101 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:13:04.872581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:04.872553 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.061285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.061183 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.183132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.183099 5733 generic.go:356] "Generic (PLEG): container finished" podID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerID="140d65a139a52a8db1def49aa9fa17d8e3b0ae77326625834c9057e339e3174e" exitCode=0 Oct 30 20:13:05.183320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.183163 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" event={"ID":"51bc38b8-b8b2-434e-819b-e7502f4d4441","Type":"ContainerDied","Data":"140d65a139a52a8db1def49aa9fa17d8e3b0ae77326625834c9057e339e3174e"} Oct 30 20:13:05.187149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.187122 5733 generic.go:356] "Generic (PLEG): container finished" podID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerID="a1306238a7d548e81974eab2be415bc43ed2aa12c82ffbb815fde40b039e1ff6" exitCode=0 Oct 30 20:13:05.187303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.187170 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" event={"ID":"78000019-d14f-4c89-b1d4-06dfcbd4be4e","Type":"ContainerDied","Data":"a1306238a7d548e81974eab2be415bc43ed2aa12c82ffbb815fde40b039e1ff6"} Oct 30 20:13:05.267957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.267919 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.464562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.464523 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.478964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.478936 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 20:13:05.507993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.507966 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 20:13:05.518176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.518147 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\" (UniqueName: \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\") pod \"51bc38b8-b8b2-434e-819b-e7502f4d4441\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " Oct 30 20:13:05.518386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.518365 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\") pod \"51bc38b8-b8b2-434e-819b-e7502f4d4441\" (UID: \"51bc38b8-b8b2-434e-819b-e7502f4d4441\") " Oct 30 20:13:05.518830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.518802 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume" pod "51bc38b8-b8b2-434e-819b-e7502f4d4441" (UID: "51bc38b8-b8b2-434e-819b-e7502f4d4441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:05.520259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.520228 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h" pod "51bc38b8-b8b2-434e-819b-e7502f4d4441" (UID: "51bc38b8-b8b2-434e-819b-e7502f4d4441"). InnerVolumeSpecName "kube-api-access-v862h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:05.620022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.619948 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\") pod \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " Oct 30 20:13:05.620196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.620139 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\" (UniqueName: \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\") pod \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\" (UID: \"78000019-d14f-4c89-b1d4-06dfcbd4be4e\") " Oct 30 20:13:05.620373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.620343 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume" pod "78000019-d14f-4c89-b1d4-06dfcbd4be4e" (UID: "78000019-d14f-4c89-b1d4-06dfcbd4be4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:05.620551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.620533 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v862h\" (UniqueName: \"kubernetes.io/projected/51bc38b8-b8b2-434e-819b-e7502f4d4441-kube-api-access-v862h\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:05.620627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.620551 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78000019-d14f-4c89-b1d4-06dfcbd4be4e-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:05.620627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.620561 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51bc38b8-b8b2-434e-819b-e7502f4d4441-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:05.622265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.622239 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn" pod "78000019-d14f-4c89-b1d4-06dfcbd4be4e" (UID: "78000019-d14f-4c89-b1d4-06dfcbd4be4e"). InnerVolumeSpecName "kube-api-access-nkwsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:05.722005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:05.721974 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkwsn\" (UniqueName: \"kubernetes.io/projected/78000019-d14f-4c89-b1d4-06dfcbd4be4e-kube-api-access-nkwsn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:06.099834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.099800 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:06.195488 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.195454 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" Oct 30 20:13:06.195679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.195494 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj" event={"ID":"51bc38b8-b8b2-434e-819b-e7502f4d4441","Type":"ContainerDied","Data":"e399d35c49300bfa2121e6eca4d5d408214c50def0bdbb492484788aff82ec37"} Oct 30 20:13:06.195679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.195564 5733 scope.go:122] "RemoveContainer" containerID="140d65a139a52a8db1def49aa9fa17d8e3b0ae77326625834c9057e339e3174e" Oct 30 20:13:06.197211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.197188 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" Oct 30 20:13:06.197329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.197185 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt" event={"ID":"78000019-d14f-4c89-b1d4-06dfcbd4be4e","Type":"ContainerDied","Data":"d105769dde36de69606017d8db29d8486dd51a3bb6750a667933f8249498da71"} Oct 30 20:13:06.206854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.206835 5733 scope.go:122] "RemoveContainer" containerID="a1306238a7d548e81974eab2be415bc43ed2aa12c82ffbb815fde40b039e1ff6" Oct 30 20:13:06.218961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.218939 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 20:13:06.224458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.224436 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-278ffa-c5cfbfd95-pbtxt"] Oct 30 20:13:06.240049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.240018 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 20:13:06.251016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.250975 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-278ffa-b9fb4886d-snxjj"] Oct 30 20:13:06.295476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:06.295440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.317990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:07.317952 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.487990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:07.487961 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.773181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:07.773148 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 20:13:07.773402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:07.773368 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" containerID="cri-o://91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de" gracePeriod=30 Oct 30 20:13:07.851293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:07.851257 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:08.049837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:08.049751 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" path="/var/lib/kubelet/pods/51bc38b8-b8b2-434e-819b-e7502f4d4441/volumes" Oct 30 20:13:08.050174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:08.050160 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" path="/var/lib/kubelet/pods/78000019-d14f-4c89-b1d4-06dfcbd4be4e/volumes" Oct 30 20:13:08.701834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:08.701795 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:08.951344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:08.951304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.117494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:09.117457 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.453753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:09.453669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.614769 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:09.614736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.919339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:09.919295 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.187531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:10.187438 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.598628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:10.598594 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.789937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:10.789906 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.970342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:10.970250 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.135251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.135211 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 20:13:11.135493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.135460 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" containerID="cri-o://e74afd51df82be2060435936a8865f5c635ccc3f2a65d4501a943bef6e90e7e0" gracePeriod=30 Oct 30 20:13:11.216015 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.215980 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.617485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.617455 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.626167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626137 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f"] Oct 30 20:13:11.626614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626601 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" Oct 30 20:13:11.626672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626615 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" Oct 30 20:13:11.626672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626638 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" Oct 30 20:13:11.626672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626645 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" Oct 30 20:13:11.626672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626656 5733 cpu_manager.go:414] "containerMap: removing container" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:11.626672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626663 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626676 5733 cpu_manager.go:414] "containerMap: removing container" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626683 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c8e7e1-6494-49a8-b61c-bb3098dcccde" containerName="maas-api" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626692 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626699 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626714 5733 cpu_manager.go:414] "containerMap: removing container" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626720 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b19494-6b48-41e6-956a-713a34b0dc10" containerName="maas-api" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626742 5733 cpu_manager.go:414] "containerMap: removing container" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626750 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626761 5733 cpu_manager.go:414] "containerMap: removing container" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626768 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626778 5733 cpu_manager.go:414] "containerMap: removing container" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626785 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626796 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5a7e7bd8-88c7-465c-be3d-fce254debc65" containerName="revoke-keys" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626801 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a7e7bd8-88c7-465c-be3d-fce254debc65" containerName="revoke-keys" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626810 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626815 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626827 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626831 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626840 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626845 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:13:11.626847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626856 5733 cpu_manager.go:414] "containerMap: removing container" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626861 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626872 5733 cpu_manager.go:414] "containerMap: removing container" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626876 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626885 5733 cpu_manager.go:414] "containerMap: removing container" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626889 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626989 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.626999 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd91d8c7-be08-4975-b46b-70f110ca1bfc" containerName="payload-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627005 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a17c23-b0b0-4bd2-a9d2-82afacf2546d" containerName="payload-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627010 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="78000019-d14f-4c89-b1d4-06dfcbd4be4e" containerName="payload-pre-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627015 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a7e7bd8-88c7-465c-be3d-fce254debc65" containerName="revoke-keys" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627021 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b700fa9e-7b5a-42f7-9d54-551c147eccc8" containerName="payload-pre-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627027 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c1b72f0-9a3d-41d2-863b-f8b2ed2b36c8" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627033 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b78a1582-b7a4-4742-a25a-94209cae3c42" containerName="istio-proxy" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627039 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e76d74-7775-4b7b-9120-885c993ad626" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627046 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="51bc38b8-b8b2-434e-819b-e7502f4d4441" containerName="payload-processing" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627052 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f65e7a-479c-44cd-9f40-5aea55deb3e1" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627079 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c5d55c-b95f-4c74-9bbb-4d660b088cfa" containerName="main" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627087 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb50a54-7c85-49dc-abfd-d9234707efd1" containerName="maas-api" Oct 30 20:13:11.627571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.627095 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d142cf77-146e-4719-8e79-001e99aead3c" containerName="main" Oct 30 20:13:11.630610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.630594 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.634824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.634799 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:11.635802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.635775 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:13:11.640243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.640219 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f"] Oct 30 20:13:11.693195 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.693158 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.693408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.693253 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.693408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.693303 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.794957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.794914 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.795185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.794962 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.795185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.794995 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.795759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.795734 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.797403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.797374 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.811362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.811338 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:11.948249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:11.948170 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:12.240882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:12.240851 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f"] Oct 30 20:13:12.242117 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:13:12.242089 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7317bbc3_d122_4cd6_a6c7_86f027d1a026.slice/crio-3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0 WatchSource:0}: Error finding container 3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0: Status 404 returned error can't find the container with id 3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0 Oct 30 20:13:12.250924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:12.250897 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" event={"ID":"7317bbc3-d122-4cd6-a6c7-86f027d1a026","Type":"ContainerStarted","Data":"3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0"} Oct 30 20:13:12.696713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:12.696672 5733 patch_prober.go:29] interesting pod/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:13:12.696915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:12.696735 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:13:12.718489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:12.718450 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.029691 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.029659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.074435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.074407 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 20:13:13.109015 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.108985 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109043 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109121 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109184 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109208 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109237 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109271 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109309 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.109643 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109451 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\") pod \"f117faf1-f824-4851-9ce6-e85ebd8fa690\" (UID: \"f117faf1-f824-4851-9ce6-e85ebd8fa690\") " Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109756 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109814 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109970 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.109989 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.110032 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:13.110080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.110049 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:13.110521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.110082 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:13.112257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.112230 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:13.112346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.112273 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:13.112346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.112291 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "kube-api-access-9hk2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:13.112515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.112490 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy" pod "f117faf1-f824-4851-9ce6-e85ebd8fa690" (UID: "f117faf1-f824-4851-9ce6-e85ebd8fa690"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:13.180152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.180118 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211110 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211151 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hk2v\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-kube-api-access-9hk2v\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211166 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211178 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f117faf1-f824-4851-9ce6-e85ebd8fa690-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211191 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211203 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.211979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.211216 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f117faf1-f824-4851-9ce6-e85ebd8fa690-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:13.260197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.260109 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" event={"ID":"7317bbc3-d122-4cd6-a6c7-86f027d1a026","Type":"ContainerStarted","Data":"fb1302058f3c36f25e860812ab6cfe25a7057ef8b36a34a03b011a184eae6dc9"} Oct 30 20:13:13.261663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.261632 5733 generic.go:356] "Generic (PLEG): container finished" podID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerID="91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de" exitCode=0 Oct 30 20:13:13.261794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.261670 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" Oct 30 20:13:13.261794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.261715 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" event={"ID":"f117faf1-f824-4851-9ce6-e85ebd8fa690","Type":"ContainerDied","Data":"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de"} Oct 30 20:13:13.261794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.261760 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb" event={"ID":"f117faf1-f824-4851-9ce6-e85ebd8fa690","Type":"ContainerDied","Data":"462d8d62658f7c098d58af4be491731d9d362d1a80ed7baaf41c3431c0fa7f2b"} Oct 30 20:13:13.261794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.261784 5733 scope.go:122] "RemoveContainer" containerID="91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de" Oct 30 20:13:13.274710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.274689 5733 scope.go:122] "RemoveContainer" containerID="91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de" Oct 30 20:13:13.275013 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:13.274995 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de\": container with ID starting with 91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de not found: ID does not exist" containerID="91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de" Oct 30 20:13:13.275185 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.275024 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de"} err="failed to get container status \"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de\": rpc error: code = NotFound desc = could not find container \"91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de\": container with ID starting with 91ab3a8717fbb3bcb0c1f2a5353348a3202e9c8c1202204a1f53b8d6ccb474de not found: ID does not exist" Oct 30 20:13:13.288574 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.288532 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" podStartSLOduration=2.288518533 podStartE2EDuration="2.288518533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:13.28822146 +0000 UTC m=+2553.356057029" watchObservedRunningTime="2026-10-30 20:13:13.288518533 +0000 UTC m=+2553.356354100" Oct 30 20:13:13.305650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.305622 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 20:13:13.313378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.313348 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-278ffa-openshift-default-579d9fbc7f-jvvhb"] Oct 30 20:13:13.343453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.343424 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.668393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:13.668354 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:14.049202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:14.049125 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" path="/var/lib/kubelet/pods/f117faf1-f824-4851-9ce6-e85ebd8fa690/volumes" Oct 30 20:13:14.271708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:14.271670 5733 generic.go:356] "Generic (PLEG): container finished" podID="7317bbc3-d122-4cd6-a6c7-86f027d1a026" containerID="fb1302058f3c36f25e860812ab6cfe25a7057ef8b36a34a03b011a184eae6dc9" exitCode=0 Oct 30 20:13:14.272139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:14.271725 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" event={"ID":"7317bbc3-d122-4cd6-a6c7-86f027d1a026","Type":"ContainerDied","Data":"fb1302058f3c36f25e860812ab6cfe25a7057ef8b36a34a03b011a184eae6dc9"} Oct 30 20:13:14.304027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:14.303928 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:15.153502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.153469 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:15.188137 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.188101 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:13:15.430012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.429986 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:15.513085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.513045 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:15.538574 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.538537 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\") pod \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " Oct 30 20:13:15.538730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.538702 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\") pod \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " Oct 30 20:13:15.538796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.538731 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\") pod \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\" (UID: \"7317bbc3-d122-4cd6-a6c7-86f027d1a026\") " Oct 30 20:13:15.539231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.539203 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca" pod "7317bbc3-d122-4cd6-a6c7-86f027d1a026" (UID: "7317bbc3-d122-4cd6-a6c7-86f027d1a026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:15.540701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.540680 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca" pod "7317bbc3-d122-4cd6-a6c7-86f027d1a026" (UID: "7317bbc3-d122-4cd6-a6c7-86f027d1a026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:15.540816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.540791 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k" pod "7317bbc3-d122-4cd6-a6c7-86f027d1a026" (UID: "7317bbc3-d122-4cd6-a6c7-86f027d1a026"). InnerVolumeSpecName "kube-api-access-wg79k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:15.640190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.640156 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg79k\" (UniqueName: \"kubernetes.io/projected/7317bbc3-d122-4cd6-a6c7-86f027d1a026-kube-api-access-wg79k\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:15.640190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.640185 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:15.640190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:15.640195 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7317bbc3-d122-4cd6-a6c7-86f027d1a026-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.289345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.289305 5733 generic.go:356] "Generic (PLEG): container finished" podID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerID="e74afd51df82be2060435936a8865f5c635ccc3f2a65d4501a943bef6e90e7e0" exitCode=0 Oct 30 20:13:16.289523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.289397 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" event={"ID":"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6","Type":"ContainerDied","Data":"e74afd51df82be2060435936a8865f5c635ccc3f2a65d4501a943bef6e90e7e0"} Oct 30 20:13:16.290918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.290887 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" Oct 30 20:13:16.291086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.290923 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f" event={"ID":"7317bbc3-d122-4cd6-a6c7-86f027d1a026","Type":"ContainerDied","Data":"3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0"} Oct 30 20:13:16.291086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.290972 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f18bc7375cfcba8d303bc3d2b5074844541b4adfe2f6d0241aa8e2ccf8eb9a0" Oct 30 20:13:16.396177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.396138 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 20:13:16.448758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.448708 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.448824 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.448870 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.448939 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.448971 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449048 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449083 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449127 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449166 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449208 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\") pod \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\" (UID: \"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6\") " Oct 30 20:13:16.449788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449325 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:16.449788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449598 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:16.449788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449651 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.449788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449672 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.450487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.449957 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:16.450681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.450656 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:16.452381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.452356 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "kube-api-access-7zs7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:16.452483 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.452408 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:16.452531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.452497 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:16.452710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.452690 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy" pod "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" (UID: "ff87fcd9-988d-4f08-aaf3-86d2ff50bab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:16.527087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.527021 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:16.550793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550760 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550791 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zs7h\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-kube-api-access-7zs7h\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550802 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550811 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550820 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550829 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:16.550998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:16.550839 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:17.299974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.299941 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" Oct 30 20:13:17.299974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.299962 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc" event={"ID":"ff87fcd9-988d-4f08-aaf3-86d2ff50bab6","Type":"ContainerDied","Data":"504b6f9d4e30c2793eb81b1df20c1639dc0bb4dde41328023499be9063e29998"} Oct 30 20:13:17.300234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.300029 5733 scope.go:122] "RemoveContainer" containerID="e74afd51df82be2060435936a8865f5c635ccc3f2a65d4501a943bef6e90e7e0" Oct 30 20:13:17.348158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.348124 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 20:13:17.362360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.362326 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-b0d13a-openshift-default-7787d68885-k2pqc"] Oct 30 20:13:17.836635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:17.836602 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:18.048971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:18.048934 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" path="/var/lib/kubelet/pods/ff87fcd9-988d-4f08-aaf3-86d2ff50bab6/volumes" Oct 30 20:13:19.008889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:19.008856 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.160234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:20.160198 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.516229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:20.516139 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.733155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:20.733102 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:21.151787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:21.151742 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:21.714610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:21.714567 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 20:13:21.971866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:21.971750 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.142999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.142945 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.327734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.327647 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.541651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.541618 5733 scope.go:122] "RemoveContainer" containerID="45fd828105b297d51743901958169d9c03629a4daec77821757ed259fa55e6e0" Oct 30 20:13:22.559522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.559490 5733 scope.go:122] "RemoveContainer" containerID="cde33313a9ac503b0a84a13efd688ee7ef3121b45f8c55a793d9e31695da2c7c" Oct 30 20:13:22.560500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.560454 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.570550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.570526 5733 scope.go:122] "RemoveContainer" containerID="fbf88ad62aa4b42357dffdb6435bcd91149b05f865c57bb788f3b4b15f37aabc" Oct 30 20:13:22.581596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.581573 5733 scope.go:122] "RemoveContainer" containerID="e9789732e4398411d3ef1eb358945c345562b92e11eb20b3d22ca454e1fa91ff" Oct 30 20:13:22.591124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.591107 5733 scope.go:122] "RemoveContainer" containerID="bdfa3d6a341557552c83656e814a421ff9e5c0179cc42cc26ac9de55cc54af46" Oct 30 20:13:22.600297 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.600279 5733 scope.go:122] "RemoveContainer" containerID="88d28d0c138dc51caa10e6e00fb5090c2eccb2357b72774451571b2f3677e47c" Oct 30 20:13:22.730515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.730488 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.836300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.836218 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 20:13:22.836507 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.836479 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" containerID="cri-o://12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635" gracePeriod=30 Oct 30 20:13:22.851210 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.851180 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 20:13:22.851391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.851372 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" containerID="cri-o://9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e" gracePeriod=30 Oct 30 20:13:22.921280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:22.921246 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:23.471322 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:23.471286 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:24.399586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:24.399550 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:25.432836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:25.432803 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:26.374622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:26.374554 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:26.715998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:26.715897 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" containerID="cri-o://8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a" gracePeriod=25 Oct 30 20:13:26.984766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:26.984736 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 20:13:27.003975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.003940 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 20:13:27.004235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.004211 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" containerID="cri-o://3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea" gracePeriod=30 Oct 30 20:13:27.065912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.065883 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\") pod \"f4c14d49-8d28-428b-80a5-51a881aae5d1\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " Oct 30 20:13:27.066099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.065947 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\" (UniqueName: \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\") pod \"f4c14d49-8d28-428b-80a5-51a881aae5d1\" (UID: \"f4c14d49-8d28-428b-80a5-51a881aae5d1\") " Oct 30 20:13:27.066294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.066270 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume" pod "f4c14d49-8d28-428b-80a5-51a881aae5d1" (UID: "f4c14d49-8d28-428b-80a5-51a881aae5d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:27.066461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.066443 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4c14d49-8d28-428b-80a5-51a881aae5d1-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.068084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.068051 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj" pod "f4c14d49-8d28-428b-80a5-51a881aae5d1" (UID: "f4c14d49-8d28-428b-80a5-51a881aae5d1"). InnerVolumeSpecName "kube-api-access-t44mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:27.102730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.102702 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.167069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.167022 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t44mj\" (UniqueName: \"kubernetes.io/projected/f4c14d49-8d28-428b-80a5-51a881aae5d1-kube-api-access-t44mj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.286333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.286307 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:13:27.362491 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.362456 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f"] Oct 30 20:13:27.371897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.371864 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-3d0a78-nxd6f"] Oct 30 20:13:27.372207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372187 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\") pod \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " Oct 30 20:13:27.372283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372248 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") pod \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " Oct 30 20:13:27.372349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372333 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\") pod \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " Oct 30 20:13:27.372422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372408 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\" (UniqueName: \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\") pod \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\" (UID: \"34580b12-bb45-4ca9-be6c-95b6421e8c0e\") " Oct 30 20:13:27.372588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372563 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca" pod "34580b12-bb45-4ca9-be6c-95b6421e8c0e" (UID: "34580b12-bb45-4ca9-be6c-95b6421e8c0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:27.372841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.372822 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/34580b12-bb45-4ca9-be6c-95b6421e8c0e-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.374535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.374508 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs" pod "34580b12-bb45-4ca9-be6c-95b6421e8c0e" (UID: "34580b12-bb45-4ca9-be6c-95b6421e8c0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:27.374637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.374578 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls" pod "34580b12-bb45-4ca9-be6c-95b6421e8c0e" (UID: "34580b12-bb45-4ca9-be6c-95b6421e8c0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:27.374683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.374653 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669" pod "34580b12-bb45-4ca9-be6c-95b6421e8c0e" (UID: "34580b12-bb45-4ca9-be6c-95b6421e8c0e"). InnerVolumeSpecName "kube-api-access-5r669". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:27.377713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.377349 5733 generic.go:356] "Generic (PLEG): container finished" podID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerID="8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a" exitCode=0 Oct 30 20:13:27.377713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.377385 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" Oct 30 20:13:27.377713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.377592 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" event={"ID":"f4c14d49-8d28-428b-80a5-51a881aae5d1","Type":"ContainerDied","Data":"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a"} Oct 30 20:13:27.377713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.377653 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw" event={"ID":"f4c14d49-8d28-428b-80a5-51a881aae5d1","Type":"ContainerDied","Data":"84bc632631a4c2019c56be2abce9766513e2ad3d20dbbe44bb7aa28b73223017"} Oct 30 20:13:27.377713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.377683 5733 scope.go:122] "RemoveContainer" containerID="8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a" Oct 30 20:13:27.379584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.379544 5733 generic.go:356] "Generic (PLEG): container finished" podID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerID="3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea" exitCode=0 Oct 30 20:13:27.379708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.379589 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" Oct 30 20:13:27.379708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.379622 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerDied","Data":"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea"} Oct 30 20:13:27.379708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.379665 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p" event={"ID":"34580b12-bb45-4ca9-be6c-95b6421e8c0e","Type":"ContainerDied","Data":"bfbd8726a1ff7919f330a1e106e438871077e33ddc5bef00ffc1f2ef255a767e"} Oct 30 20:13:27.383372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.383345 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.388637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.388617 5733 scope.go:122] "RemoveContainer" containerID="8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a" Oct 30 20:13:27.388995 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:27.388894 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a\": container with ID starting with 8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a not found: ID does not exist" containerID="8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a" Oct 30 20:13:27.388995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.388922 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a"} err="failed to get container status \"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a\": rpc error: code = NotFound desc = could not find container \"8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a\": container with ID starting with 8e4a58a7d02baef04950fdd76b7b799ecda193706d7cb618e6dbb5352047ec1a not found: ID does not exist" Oct 30 20:13:27.388995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.388945 5733 scope.go:122] "RemoveContainer" containerID="3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea" Oct 30 20:13:27.398949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.398929 5733 scope.go:122] "RemoveContainer" containerID="282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75" Oct 30 20:13:27.408209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.408189 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 20:13:27.408383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.408370 5733 scope.go:122] "RemoveContainer" containerID="3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea" Oct 30 20:13:27.408672 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:27.408645 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea\": container with ID starting with 3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea not found: ID does not exist" containerID="3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea" Oct 30 20:13:27.408781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.408680 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea"} err="failed to get container status \"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea\": rpc error: code = NotFound desc = could not find container \"3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea\": container with ID starting with 3abeb2ded8d6b56cbe3db64c18b64cdf51971a0451898205e71009c01033b9ea not found: ID does not exist" Oct 30 20:13:27.408781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.408703 5733 scope.go:122] "RemoveContainer" containerID="282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75" Oct 30 20:13:27.409163 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:27.409137 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75\": container with ID starting with 282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75 not found: ID does not exist" containerID="282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75" Oct 30 20:13:27.409249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.409172 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75"} err="failed to get container status \"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75\": rpc error: code = NotFound desc = could not find container \"282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75\": container with ID starting with 282a5955dd4937e6b2ad905dbea074f8d0f00c41b91c6561f30386a0077b3a75 not found: ID does not exist" Oct 30 20:13:27.414437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.414408 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3d0a78-bcf567d44-tb6nw"] Oct 30 20:13:27.442872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.442843 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 20:13:27.459574 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.459547 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-3d0a78-66775c7ff4-xdj9p"] Oct 30 20:13:27.474257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.474225 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.474257 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.474257 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/34580b12-bb45-4ca9-be6c-95b6421e8c0e-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.474434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.474275 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r669\" (UniqueName: \"kubernetes.io/projected/34580b12-bb45-4ca9-be6c-95b6421e8c0e-kube-api-access-5r669\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.558342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.558250 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.965866 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:27.965840 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:28.053712 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.053662 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" path="/var/lib/kubelet/pods/34580b12-bb45-4ca9-be6c-95b6421e8c0e/volumes" Oct 30 20:13:28.059190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.059160 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7317bbc3-d122-4cd6-a6c7-86f027d1a026" path="/var/lib/kubelet/pods/7317bbc3-d122-4cd6-a6c7-86f027d1a026/volumes" Oct 30 20:13:28.059750 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.059729 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" path="/var/lib/kubelet/pods/f4c14d49-8d28-428b-80a5-51a881aae5d1/volumes" Oct 30 20:13:28.102953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.102927 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 20:13:28.140333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.140306 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 20:13:28.182764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182734 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.182921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182809 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.182921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182843 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.182921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182886 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.182921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182919 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.183175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182955 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.183175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.182997 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.183175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183065 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.183175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183091 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.183175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183149 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\") pod \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\" (UID: \"b535075b-1639-47bb-b428-7c5c0eb5d4b2\") " Oct 30 20:13:28.183447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183306 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.183529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183500 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.183529 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183503 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:28.183626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183590 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.183626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183611 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.184018 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.183976 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.185407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.185376 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:28.185603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.185577 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "kube-api-access-flzs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.185688 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.185635 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.185790 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.185770 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token" pod "b535075b-1639-47bb-b428-7c5c0eb5d4b2" (UID: "b535075b-1639-47bb-b428-7c5c0eb5d4b2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.284205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284168 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284209 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284245 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284280 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284338 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284462 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284563 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284619 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.284744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284657 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.284744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284656 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.284744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284713 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:28.284744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284717 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\") pod \"0f49df1b-29d9-48cc-b270-00a78df3439e\" (UID: \"0f49df1b-29d9-48cc-b270-00a78df3439e\") " Oct 30 20:13:28.285026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.284897 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.285131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.285203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285180 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285196 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzs5\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-kube-api-access-flzs5\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285206 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285215 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285224 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285233 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285241 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285250 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f49df1b-29d9-48cc-b270-00a78df3439e-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285263 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285276 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b535075b-1639-47bb-b428-7c5c0eb5d4b2-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285287 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.285310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.285300 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b535075b-1639-47bb-b428-7c5c0eb5d4b2-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.286772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.286750 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:28.287035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.287010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.287035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.287021 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:28.287202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.287033 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2" pod "0f49df1b-29d9-48cc-b270-00a78df3439e" (UID: "0f49df1b-29d9-48cc-b270-00a78df3439e"). InnerVolumeSpecName "kube-api-access-s7sv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.386026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.385941 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.386026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.385969 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.386026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.385980 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7sv2\" (UniqueName: \"kubernetes.io/projected/0f49df1b-29d9-48cc-b270-00a78df3439e-kube-api-access-s7sv2\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.386026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.385991 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f49df1b-29d9-48cc-b270-00a78df3439e-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.389203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.389172 5733 generic.go:356] "Generic (PLEG): container finished" podID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerID="12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635" exitCode=0 Oct 30 20:13:28.389354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.389221 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" Oct 30 20:13:28.389354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.389256 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" event={"ID":"b535075b-1639-47bb-b428-7c5c0eb5d4b2","Type":"ContainerDied","Data":"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635"} Oct 30 20:13:28.389354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.389306 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4" event={"ID":"b535075b-1639-47bb-b428-7c5c0eb5d4b2","Type":"ContainerDied","Data":"bdc914c13504880180034edbcca0cc9929926bb2e194b7e03dadc749de24380e"} Oct 30 20:13:28.389354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.389332 5733 scope.go:122] "RemoveContainer" containerID="12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635" Oct 30 20:13:28.391197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.391174 5733 generic.go:356] "Generic (PLEG): container finished" podID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerID="9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e" exitCode=0 Oct 30 20:13:28.391317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.391222 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" Oct 30 20:13:28.391317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.391258 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerDied","Data":"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e"} Oct 30 20:13:28.391317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.391312 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72" event={"ID":"0f49df1b-29d9-48cc-b270-00a78df3439e","Type":"ContainerDied","Data":"caf43166f3c3ae8de65273cab246653177b029cc696ac7297f17fb5ff8a17ecb"} Oct 30 20:13:28.397540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.397488 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:28.403213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.403180 5733 scope.go:122] "RemoveContainer" containerID="12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635" Oct 30 20:13:28.403508 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:28.403483 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635\": container with ID starting with 12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635 not found: ID does not exist" containerID="12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635" Oct 30 20:13:28.403609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.403515 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635"} err="failed to get container status \"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635\": rpc error: code = NotFound desc = could not find container \"12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635\": container with ID starting with 12f9987ffbcb79a04b570b9b2692f6f4f8974535d9d2afd0a9e3b9e419c41635 not found: ID does not exist" Oct 30 20:13:28.403609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.403533 5733 scope.go:122] "RemoveContainer" containerID="9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e" Oct 30 20:13:28.414595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.414569 5733 scope.go:122] "RemoveContainer" containerID="7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1" Oct 30 20:13:28.422150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.422126 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 20:13:28.428823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.428798 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-kwl72"] Oct 30 20:13:28.429888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.429847 5733 scope.go:122] "RemoveContainer" containerID="9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e" Oct 30 20:13:28.430235 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:28.430213 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e\": container with ID starting with 9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e not found: ID does not exist" containerID="9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e" Oct 30 20:13:28.430362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.430253 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e"} err="failed to get container status \"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e\": rpc error: code = NotFound desc = could not find container \"9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e\": container with ID starting with 9e99f51add51bc3beb058b67e58fc8879edef25af175d88e7effb5e15fa18c7e not found: ID does not exist" Oct 30 20:13:28.430362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.430277 5733 scope.go:122] "RemoveContainer" containerID="7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1" Oct 30 20:13:28.430579 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:28.430560 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1\": container with ID starting with 7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1 not found: ID does not exist" containerID="7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1" Oct 30 20:13:28.430624 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.430583 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1"} err="failed to get container status \"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1\": rpc error: code = NotFound desc = could not find container \"7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1\": container with ID starting with 7af238a535c1f101ae33a290da249945aff7e2944f22c36b05c2e948aa0e07f1 not found: ID does not exist" Oct 30 20:13:28.438679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.438656 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 20:13:28.444081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:28.444045 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-5f933e-openshift-default-58868fc4f5-g9mv4"] Oct 30 20:13:29.378077 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:29.378024 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:13:29.486554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:29.486515 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:30.048901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:30.048856 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" path="/var/lib/kubelet/pods/0f49df1b-29d9-48cc-b270-00a78df3439e/volumes" Oct 30 20:13:30.049439 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:30.049415 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" path="/var/lib/kubelet/pods/b535075b-1639-47bb-b428-7c5c0eb5d4b2/volumes" Oct 30 20:13:30.419963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:30.419927 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:31.580428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:31.580397 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.564422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:32.564393 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:33.665780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:33.665748 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:34.542976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:34.542943 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:35.601526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:35.601491 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:36.619357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:36.619326 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.697550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:37.697520 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.978991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:37.978905 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.151598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.151558 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.409597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.409559 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.680760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.680667 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.843901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.843868 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.955199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955118 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk"] Oct 30 20:13:38.955551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955538 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:38.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955553 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:38.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955574 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" Oct 30 20:13:38.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955579 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" Oct 30 20:13:38.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955587 5733 cpu_manager.go:414] "containerMap: removing container" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:38.955598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955592 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955607 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955611 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955619 5733 cpu_manager.go:414] "containerMap: removing container" podUID="7317bbc3-d122-4cd6-a6c7-86f027d1a026" containerName="revoke-keys" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955624 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7317bbc3-d122-4cd6-a6c7-86f027d1a026" containerName="revoke-keys" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955633 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955637 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955646 5733 cpu_manager.go:414] "containerMap: removing container" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955650 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955660 5733 cpu_manager.go:414] "containerMap: removing container" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955664 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955679 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955684 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955742 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff87fcd9-988d-4f08-aaf3-86d2ff50bab6" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955751 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="7317bbc3-d122-4cd6-a6c7-86f027d1a026" containerName="revoke-keys" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955758 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955765 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="b535075b-1639-47bb-b428-7c5c0eb5d4b2" containerName="istio-proxy" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955771 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:38.955775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955778 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="affa7e70-81df-440e-a795-9e0b5ee3ae27" containerName="istio-proxy" Oct 30 20:13:38.956384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955786 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4c14d49-8d28-428b-80a5-51a881aae5d1" containerName="payload-processing" Oct 30 20:13:38.956384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.955794 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f117faf1-f824-4851-9ce6-e85ebd8fa690" containerName="istio-proxy" Oct 30 20:13:38.959266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.959248 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:38.973179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.973156 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:38.973311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.973242 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:38.975140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.975117 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk"] Oct 30 20:13:38.998710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.998680 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:38.998854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.998718 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcfwz\" (UniqueName: \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:38.998903 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:38.998856 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.045166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.045134 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:39.045358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.045175 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:13:39.099481 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.099445 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.099481 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.099485 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcfwz\" (UniqueName: \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.099677 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.099558 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.100109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.100080 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.101915 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.101894 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.112799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.112769 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcfwz\" (UniqueName: \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\") pod \"maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.272898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.272802 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:39.555466 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.555434 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk"] Oct 30 20:13:39.560215 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:13:39.560184 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74648165_634c_47ad_bda2_a591786b61a8.slice/crio-cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb WatchSource:0}: Error finding container cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb: Status 404 returned error can't find the container with id cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb Oct 30 20:13:39.913777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:39.913716 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:40.486924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:40.486897 5733 generic.go:356] "Generic (PLEG): container finished" podID="74648165-634c-47ad-bda2-a591786b61a8" containerID="c2567aea3b63134b5a9ab548fb4dced0a8c4cd240cb0eb95b81a266a368fcaaf" exitCode=0 Oct 30 20:13:40.487052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:40.486949 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" event={"ID":"74648165-634c-47ad-bda2-a591786b61a8","Type":"ContainerDied","Data":"c2567aea3b63134b5a9ab548fb4dced0a8c4cd240cb0eb95b81a266a368fcaaf"} Oct 30 20:13:40.487052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:40.487011 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" event={"ID":"74648165-634c-47ad-bda2-a591786b61a8","Type":"ContainerStarted","Data":"cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb"} Oct 30 20:13:40.717267 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:40.717173 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:41.578540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.578475 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:41.637557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.637534 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:41.725814 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.725781 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\") pod \"74648165-634c-47ad-bda2-a591786b61a8\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " Oct 30 20:13:41.726030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.725854 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\") pod \"74648165-634c-47ad-bda2-a591786b61a8\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " Oct 30 20:13:41.726030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.725910 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\" (UniqueName: \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\") pod \"74648165-634c-47ad-bda2-a591786b61a8\" (UID: \"74648165-634c-47ad-bda2-a591786b61a8\") " Oct 30 20:13:41.726252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.726223 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca" pod "74648165-634c-47ad-bda2-a591786b61a8" (UID: "74648165-634c-47ad-bda2-a591786b61a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:41.728107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.728083 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz" pod "74648165-634c-47ad-bda2-a591786b61a8" (UID: "74648165-634c-47ad-bda2-a591786b61a8"). InnerVolumeSpecName "kube-api-access-mcfwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:41.728107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.728088 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca" pod "74648165-634c-47ad-bda2-a591786b61a8" (UID: "74648165-634c-47ad-bda2-a591786b61a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:41.826819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.826751 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcfwz\" (UniqueName: \"kubernetes.io/projected/74648165-634c-47ad-bda2-a591786b61a8-kube-api-access-mcfwz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:41.826819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.826778 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74648165-634c-47ad-bda2-a591786b61a8-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:41.826819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:41.826789 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74648165-634c-47ad-bda2-a591786b61a8-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:42.429503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:42.429463 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:42.501644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:42.501596 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" event={"ID":"74648165-634c-47ad-bda2-a591786b61a8","Type":"ContainerDied","Data":"cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb"} Oct 30 20:13:42.501644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:42.501617 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk" Oct 30 20:13:42.501644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:42.501648 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfeb28125a2da1d11d29c2a4c33247b23d904da26497663e32271e954c3c9fb" Oct 30 20:13:43.189440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:43.189409 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:43.976258 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:43.976228 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:44.128400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:44.128368 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:44.966856 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:44.966820 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.135946 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:45.135906 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.275286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:45.275198 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.692761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:45.692697 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:46.577618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:46.577590 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:47.398867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:47.398796 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:48.295446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:48.295408 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:49.011606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:49.011575 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 20:13:49.090798 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:49.090761 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:49.264512 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:49.264430 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:49.949182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:49.949147 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:50.807880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:50.807833 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:51.686196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:51.686165 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.294005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.293969 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d"] Oct 30 20:13:52.294448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294433 5733 cpu_manager.go:414] "containerMap: removing container" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:52.294510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294450 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:52.294510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294482 5733 cpu_manager.go:414] "containerMap: removing container" podUID="74648165-634c-47ad-bda2-a591786b61a8" containerName="revoke-keys" Oct 30 20:13:52.294510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294488 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74648165-634c-47ad-bda2-a591786b61a8" containerName="revoke-keys" Oct 30 20:13:52.294619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294564 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f49df1b-29d9-48cc-b270-00a78df3439e" containerName="istio-proxy" Oct 30 20:13:52.294619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294577 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="34580b12-bb45-4ca9-be6c-95b6421e8c0e" containerName="maas-api" Oct 30 20:13:52.294619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.294589 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="74648165-634c-47ad-bda2-a591786b61a8" containerName="revoke-keys" Oct 30 20:13:52.298143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.298123 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.304599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.304576 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:13:52.304735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.304668 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:13:52.304805 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.304747 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:13:52.311036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.311009 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d"] Oct 30 20:13:52.311169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.311135 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:13:52.438001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.437951 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.438171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.438015 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.438171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.438137 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sk6\" (UniqueName: \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.539823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.539788 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.539823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.539833 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.540044 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.539854 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sk6\" (UniqueName: \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.540485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.540452 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.542366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.542337 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.552236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.552178 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sk6\" (UniqueName: \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\") pod \"maas-api-revoke-sub-0d79c748-fh88d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.626172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.626135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.632982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.632951 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:52.765535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.765503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.914838 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:52.914810 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d"] Oct 30 20:13:52.917017 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:13:52.916989 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee3e548_8002_448c_b440_b81408fef84d.slice/crio-d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9 WatchSource:0}: Error finding container d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9: Status 404 returned error can't find the container with id d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9 Oct 30 20:13:53.484368 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:53.484284 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.576943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:53.576909 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" event={"ID":"5ee3e548-8002-448c-b440-b81408fef84d","Type":"ContainerStarted","Data":"7e56c72d18623217ef13e4f878c1c19ae981ab9d074d81c81db68c70fd33aede"} Oct 30 20:13:53.577142 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:53.576967 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" event={"ID":"5ee3e548-8002-448c-b440-b81408fef84d","Type":"ContainerStarted","Data":"d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9"} Oct 30 20:13:53.595957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:53.595910 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" podStartSLOduration=1.595896386 podStartE2EDuration="1.595896386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:13:53.593910032 +0000 UTC m=+2593.661745601" watchObservedRunningTime="2026-10-30 20:13:53.595896386 +0000 UTC m=+2593.663731953" Oct 30 20:13:54.013110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:54.013052 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" containerID="cri-o://a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b" gracePeriod=25 Oct 30 20:13:54.071831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:54.071803 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:54.295871 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:54.295791 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:54.586182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:54.586096 5733 generic.go:356] "Generic (PLEG): container finished" podID="5ee3e548-8002-448c-b440-b81408fef84d" containerID="7e56c72d18623217ef13e4f878c1c19ae981ab9d074d81c81db68c70fd33aede" exitCode=0 Oct 30 20:13:54.586338 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:54.586179 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" event={"ID":"5ee3e548-8002-448c-b440-b81408fef84d","Type":"ContainerDied","Data":"7e56c72d18623217ef13e4f878c1c19ae981ab9d074d81c81db68c70fd33aede"} Oct 30 20:13:55.190593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.190563 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:55.733455 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.733428 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:55.874407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.874376 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\") pod \"5ee3e548-8002-448c-b440-b81408fef84d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " Oct 30 20:13:55.874582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.874565 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\") pod \"5ee3e548-8002-448c-b440-b81408fef84d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " Oct 30 20:13:55.874645 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.874635 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\" (UniqueName: \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\") pod \"5ee3e548-8002-448c-b440-b81408fef84d\" (UID: \"5ee3e548-8002-448c-b440-b81408fef84d\") " Oct 30 20:13:55.874921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.874891 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca" pod "5ee3e548-8002-448c-b440-b81408fef84d" (UID: "5ee3e548-8002-448c-b440-b81408fef84d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:55.875082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.875022 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:55.876710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.876683 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca" pod "5ee3e548-8002-448c-b440-b81408fef84d" (UID: "5ee3e548-8002-448c-b440-b81408fef84d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:13:55.876888 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.876814 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6" pod "5ee3e548-8002-448c-b440-b81408fef84d" (UID: "5ee3e548-8002-448c-b440-b81408fef84d"). InnerVolumeSpecName "kube-api-access-86sk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:55.976341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.976306 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86sk6\" (UniqueName: \"kubernetes.io/projected/5ee3e548-8002-448c-b440-b81408fef84d-kube-api-access-86sk6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:55.976341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.976339 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ee3e548-8002-448c-b440-b81408fef84d-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:55.991294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:55.991259 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:56.600036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:56.600008 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" Oct 30 20:13:56.600549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:56.600007 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d" event={"ID":"5ee3e548-8002-448c-b440-b81408fef84d","Type":"ContainerDied","Data":"d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9"} Oct 30 20:13:56.600549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:56.600136 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5931913d6d363947f25a051a8f9a5af7b44aadbe0a78e2aefbbbaebdc83b8f9" Oct 30 20:13:56.795893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:56.795854 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:57.799541 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:57.799506 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:58.898181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:58.898142 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.158201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:59.158114 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:13:59.377749 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:13:59.377714 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:13:59.685632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:13:59.685600 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:00.575621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:00.575589 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:01.460457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:01.460419 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.282834 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:02.282799 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:03.079549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:03.079510 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:03.955609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:03.955574 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:04.694084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:04.694017 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.556458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:05.556425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:06.296846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:06.296810 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.209765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:07.209736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.411080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:07.411026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:08.003092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:08.003028 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:08.923877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:08.923841 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:09.158635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:09.158592 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:14:09.810314 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:09.810269 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:09.999511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:09.999478 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.068904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:10.068806 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk"] Oct 30 20:14:10.072586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:10.072558 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-ced832-jm7mk"] Oct 30 20:14:10.272862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:10.272829 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.473989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:10.473960 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.654924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:10.654887 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:11.212236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:11.212204 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:11.999327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:11.999295 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.050271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:12.050239 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74648165-634c-47ad-bda2-a591786b61a8" path="/var/lib/kubelet/pods/74648165-634c-47ad-bda2-a591786b61a8/volumes" Oct 30 20:14:12.787407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:12.787374 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:13.583386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:13.583356 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:14.284687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.284616 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 20:14:14.367303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.367261 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\") pod \"88acb5d2-4a06-4b9f-a408-0137dad64956\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " Oct 30 20:14:14.367516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.367343 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\" (UniqueName: \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\") pod \"88acb5d2-4a06-4b9f-a408-0137dad64956\" (UID: \"88acb5d2-4a06-4b9f-a408-0137dad64956\") " Oct 30 20:14:14.367659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.367630 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume" pod "88acb5d2-4a06-4b9f-a408-0137dad64956" (UID: "88acb5d2-4a06-4b9f-a408-0137dad64956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:14.369510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.369479 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh" pod "88acb5d2-4a06-4b9f-a408-0137dad64956" (UID: "88acb5d2-4a06-4b9f-a408-0137dad64956"). InnerVolumeSpecName "kube-api-access-vmssh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:14.468351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.468308 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acb5d2-4a06-4b9f-a408-0137dad64956-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.468351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.468342 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmssh\" (UniqueName: \"kubernetes.io/projected/88acb5d2-4a06-4b9f-a408-0137dad64956-kube-api-access-vmssh\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:14.582281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.582174 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:14.727413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.727380 5733 generic.go:356] "Generic (PLEG): container finished" podID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerID="a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b" exitCode=0 Oct 30 20:14:14.727607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.727449 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" Oct 30 20:14:14.727607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.727465 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" event={"ID":"88acb5d2-4a06-4b9f-a408-0137dad64956","Type":"ContainerDied","Data":"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b"} Oct 30 20:14:14.727607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.727540 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx" event={"ID":"88acb5d2-4a06-4b9f-a408-0137dad64956","Type":"ContainerDied","Data":"498ac3bb8d0ab48ce716b6fccd470e0584900fc258af744d78629a1185e1f33a"} Oct 30 20:14:14.727607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.727579 5733 scope.go:122] "RemoveContainer" containerID="a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b" Oct 30 20:14:14.739468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.739438 5733 scope.go:122] "RemoveContainer" containerID="a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b" Oct 30 20:14:14.739881 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:14:14.739841 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b\": container with ID starting with a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b not found: ID does not exist" containerID="a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b" Oct 30 20:14:14.740011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.739879 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b"} err="failed to get container status \"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b\": rpc error: code = NotFound desc = could not find container \"a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b\": container with ID starting with a47cdde03d731b30a97fd6eb86712d2783a85b26ed8014be7f37189a3aad376b not found: ID does not exist" Oct 30 20:14:14.751563 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.751542 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 20:14:14.757077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:14.757042 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-ced832-778596c5cf-hjrqx"] Oct 30 20:14:15.571561 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:15.571532 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:16.048476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:16.048430 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" path="/var/lib/kubelet/pods/88acb5d2-4a06-4b9f-a408-0137dad64956/volumes" Oct 30 20:14:16.474069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:16.474018 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:17.074152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074112 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:14:17.074581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074566 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5ee3e548-8002-448c-b440-b81408fef84d" containerName="revoke-keys" Oct 30 20:14:17.074623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074582 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee3e548-8002-448c-b440-b81408fef84d" containerName="revoke-keys" Oct 30 20:14:17.074623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074616 5733 cpu_manager.go:414] "containerMap: removing container" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" Oct 30 20:14:17.074696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074622 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" Oct 30 20:14:17.074732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074709 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee3e548-8002-448c-b440-b81408fef84d" containerName="revoke-keys" Oct 30 20:14:17.074732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.074724 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="88acb5d2-4a06-4b9f-a408-0137dad64956" containerName="payload-pre-processing" Oct 30 20:14:17.078978 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.078961 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.097977 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.097949 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae2147-openshift-default-dockercfg-x9mn6\"" Oct 30 20:14:17.110437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.110406 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:14:17.126094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.126030 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:17.130541 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.130513 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.147474 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.147442 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:14:17.151731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.151709 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.194108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.194078 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:17.196212 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196187 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.196350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196235 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.196350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196280 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.196350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196313 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.196350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196336 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196359 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196381 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196402 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196423 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlwz\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196448 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196463 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.196606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196540 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.196900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196656 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.196900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196751 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.196900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196779 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsrr\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.196900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196805 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.197123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196927 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.197123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.196968 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.197123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197006 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpwgx\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.197280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197128 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.197280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197166 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.197280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197209 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.197453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197283 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.197453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197329 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.197453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197359 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.197453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197401 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.197453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.197442 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.208511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.208475 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:17.208837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.208809 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:14:17.298753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298717 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.298941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298774 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.298941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298799 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.298941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298823 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpwgx\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.298941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298846 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298943 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.298988 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299125 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299168 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299206 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299249 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299279 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299304 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299321 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299359 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299394 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299446 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299484 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299514 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299519 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299532 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299562 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299606 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299624 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299649 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlwz\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299687 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299712 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299739 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299774 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299815 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299842 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsrr\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299860 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.299940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299893 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300000 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.299687 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300296 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300447 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300447 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300674 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300688 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.300818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300777 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.301308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.300914 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.302468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.302441 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.302627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.302607 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.302795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.302776 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.303105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.303077 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.303236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.303214 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.303293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.303279 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.366639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.366604 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.374545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.374512 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlwz\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.389333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.389304 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsrr\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.389526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.389503 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpwgx\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.389957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.389933 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\") pod \"e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.390390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.390366 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.392425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.392406 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:17.448794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.448747 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:17.470773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.466877 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:17.564705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.564624 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 20:14:17.564893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.564853 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" containerID="cri-o://ea53bfaf7fa074f187adf9ab5649a7b4786373a9fbeb6dc775ecb810ea2c2df3" gracePeriod=30 Oct 30 20:14:17.569391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.569361 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:17.619636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.619560 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:17.727765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.727739 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:14:17.729722 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:17.729690 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2cc347_b3e0_414b_b67a_d4444cfcc575.slice/crio-8c02d6570877302173fe8f7c4e05d4c365d19412a65eb20e27003653f0b5e103 WatchSource:0}: Error finding container 8c02d6570877302173fe8f7c4e05d4c365d19412a65eb20e27003653f0b5e103: Status 404 returned error can't find the container with id 8c02d6570877302173fe8f7c4e05d4c365d19412a65eb20e27003653f0b5e103 Oct 30 20:14:17.732437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.732404 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.732530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.732485 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.732572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.732533 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.766098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.766041 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" event={"ID":"ed2cc347-b3e0-414b-b67a-d4444cfcc575","Type":"ContainerStarted","Data":"8c02d6570877302173fe8f7c4e05d4c365d19412a65eb20e27003653f0b5e103"} Oct 30 20:14:17.837372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.837337 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:17.838207 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:17.838175 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b18ac19_913d_42bc_bcad_a2ea9bc26439.slice/crio-a746ea1d86c4b53a1ba4638fd9a183e7f4f62bc74021bf8027d3b064b94291af WatchSource:0}: Error finding container a746ea1d86c4b53a1ba4638fd9a183e7f4f62bc74021bf8027d3b064b94291af: Status 404 returned error can't find the container with id a746ea1d86c4b53a1ba4638fd9a183e7f4f62bc74021bf8027d3b064b94291af Oct 30 20:14:17.841290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.841253 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.841401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.841338 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.841401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.841384 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.859663 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.859632 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:14:17.865725 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:17.865693 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc883896_81be_4f04_8855_26253c0e7055.slice/crio-07cbe324b62c793986a60a8f9b9d3fc9847cfa11a89af58fe418b550d2215fa7 WatchSource:0}: Error finding container 07cbe324b62c793986a60a8f9b9d3fc9847cfa11a89af58fe418b550d2215fa7: Status 404 returned error can't find the container with id 07cbe324b62c793986a60a8f9b9d3fc9847cfa11a89af58fe418b550d2215fa7 Oct 30 20:14:17.867924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.867883 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.868045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.867965 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:17.868045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:17.868007 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:18.038897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.038865 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:18.774647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.774606 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" event={"ID":"cc883896-81be-4f04-8855-26253c0e7055","Type":"ContainerStarted","Data":"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3"} Oct 30 20:14:18.775147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.774658 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" event={"ID":"cc883896-81be-4f04-8855-26253c0e7055","Type":"ContainerStarted","Data":"07cbe324b62c793986a60a8f9b9d3fc9847cfa11a89af58fe418b550d2215fa7"} Oct 30 20:14:18.776902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.776838 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" event={"ID":"ed2cc347-b3e0-414b-b67a-d4444cfcc575","Type":"ContainerStarted","Data":"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774"} Oct 30 20:14:18.778622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.778603 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" event={"ID":"5b18ac19-913d-42bc-bcad-a2ea9bc26439","Type":"ContainerStarted","Data":"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9"} Oct 30 20:14:18.778749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.778636 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" event={"ID":"5b18ac19-913d-42bc-bcad-a2ea9bc26439","Type":"ContainerStarted","Data":"a746ea1d86c4b53a1ba4638fd9a183e7f4f62bc74021bf8027d3b064b94291af"} Oct 30 20:14:18.778749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.778714 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" podUID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" containerName="istio-proxy" containerID="cri-o://626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9" gracePeriod=30 Oct 30 20:14:18.858080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.845701 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podStartSLOduration=1.845681519 podStartE2EDuration="1.845681519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:18.825486259 +0000 UTC m=+2618.893321893" watchObservedRunningTime="2026-10-30 20:14:18.845681519 +0000 UTC m=+2618.913517088" Oct 30 20:14:18.882527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.882466 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podStartSLOduration=1.882449093 podStartE2EDuration="1.882449093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:18.874475586 +0000 UTC m=+2618.942311153" watchObservedRunningTime="2026-10-30 20:14:18.882449093 +0000 UTC m=+2618.950284680" Oct 30 20:14:18.886187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:18.886162 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:19.393249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.393205 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:19.394556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.394518 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 20:14:19.394747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.394728 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 20:14:19.469984 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.469254 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:19.470585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.470564 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 20:14:19.470741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.470721 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 20:14:19.926179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:19.926146 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:20.040085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.038141 5733 patch_prober.go:29] interesting pod/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:14:20.040085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.038200 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:14:20.393249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.393216 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 20:14:20.393464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.393270 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 20:14:20.475127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.469435 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 20:14:20.475127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.469498 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 20:14:20.838153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:20.835859 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:21.085459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.085073 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:21.232001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.231963 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:21.393802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.393765 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 20:14:21.393981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.393828 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 20:14:21.468171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.468129 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 20:14:21.468359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.468189 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 20:14:21.836891 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:21.836804 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:22.393074 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.393021 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 20:14:22.393590 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.393092 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 20:14:22.467420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.467374 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 20:14:22.467632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.467428 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 20:14:22.811078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.809226 5733 generic.go:356] "Generic (PLEG): container finished" podID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerID="ea53bfaf7fa074f187adf9ab5649a7b4786373a9fbeb6dc775ecb810ea2c2df3" exitCode=0 Oct 30 20:14:22.811078 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.809292 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" event={"ID":"9db2cd8e-f002-40ae-9ca1-66041fdec5bd","Type":"ContainerDied","Data":"ea53bfaf7fa074f187adf9ab5649a7b4786373a9fbeb6dc775ecb810ea2c2df3"} Oct 30 20:14:22.843334 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:22.833879 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:23.076352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.076317 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 20:14:23.132083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.129379 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" podStartSLOduration=6.129353922 podStartE2EDuration="6.129353922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:18.936220192 +0000 UTC m=+2619.004055761" watchObservedRunningTime="2026-10-30 20:14:23.129353922 +0000 UTC m=+2623.197189491" Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155472 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155535 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155604 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155647 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155676 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155731 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155769 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155804 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.155925 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\") pod \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\" (UID: \"9db2cd8e-f002-40ae-9ca1-66041fdec5bd\") " Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.157450 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.157775 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.160010 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.160099 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:23.160182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.160153 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:23.181098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.180320 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "kube-api-access-m4xlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:23.181098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.180943 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:23.181098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.181020 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:23.182530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.182499 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy" pod "9db2cd8e-f002-40ae-9ca1-66041fdec5bd" (UID: "9db2cd8e-f002-40ae-9ca1-66041fdec5bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260796 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xlb\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-kube-api-access-m4xlb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260842 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260858 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260871 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260885 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260899 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260912 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260925 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.261020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.260939 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db2cd8e-f002-40ae-9ca1-66041fdec5bd-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:23.392912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.392879 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 20:14:23.393117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.392936 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 20:14:23.471083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.470411 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Oct 30 20:14:23.471083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.470474 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Oct 30 20:14:23.850799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.848265 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" event={"ID":"9db2cd8e-f002-40ae-9ca1-66041fdec5bd","Type":"ContainerDied","Data":"4c591954c398f61cd32efdc1aaa47886b028a283c5f5760f2d1e8049601dde53"} Oct 30 20:14:23.850799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.848342 5733 scope.go:122] "RemoveContainer" containerID="ea53bfaf7fa074f187adf9ab5649a7b4786373a9fbeb6dc775ecb810ea2c2df3" Oct 30 20:14:23.850799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.848525 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6" Oct 30 20:14:23.903103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.903074 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 20:14:23.935080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:23.934861 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-rnhl6"] Oct 30 20:14:24.053410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.053307 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" path="/var/lib/kubelet/pods/9db2cd8e-f002-40ae-9ca1-66041fdec5bd/volumes" Oct 30 20:14:24.063859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.063509 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.083449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.083421 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:24.174373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174335 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174379 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174442 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174518 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174547 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174586 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174639 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174973 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174688 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174973 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174749 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\") pod \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\" (UID: \"5b18ac19-913d-42bc-bcad-a2ea9bc26439\") " Oct 30 20:14:24.174973 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.174754 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:24.175206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175013 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:24.175206 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175030 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:24.175389 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175245 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:24.175476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175444 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.175549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175471 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.175549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175536 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.175679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175548 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.175679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.175556 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:24.177236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.177207 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:24.177497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.177468 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "kube-api-access-jpwgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:24.177635 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.177612 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:24.177701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.177655 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo" pod "5b18ac19-913d-42bc-bcad-a2ea9bc26439" (UID: "5b18ac19-913d-42bc-bcad-a2ea9bc26439"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:24.243629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.243580 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.277165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.277116 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.277165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.277167 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpwgx\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-kube-api-access-jpwgx\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.277410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.277189 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.277410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.277205 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.277410 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.277220 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b18ac19-913d-42bc-bcad-a2ea9bc26439-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:24.446551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.446454 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:24.447558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.447500 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:24.448620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.448596 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:14:24.507107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.507015 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:24.507995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.507964 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:24.509506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.509486 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:14:24.857037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.856998 5733 generic.go:356] "Generic (PLEG): container finished" podID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" containerID="626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9" exitCode=0 Oct 30 20:14:24.857272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.857052 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" Oct 30 20:14:24.857272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.857152 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" event={"ID":"5b18ac19-913d-42bc-bcad-a2ea9bc26439","Type":"ContainerDied","Data":"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9"} Oct 30 20:14:24.857272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.857216 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx" event={"ID":"5b18ac19-913d-42bc-bcad-a2ea9bc26439","Type":"ContainerDied","Data":"a746ea1d86c4b53a1ba4638fd9a183e7f4f62bc74021bf8027d3b064b94291af"} Oct 30 20:14:24.857272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.857244 5733 scope.go:122] "RemoveContainer" containerID="626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9" Oct 30 20:14:24.871875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.871858 5733 scope.go:122] "RemoveContainer" containerID="626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9" Oct 30 20:14:24.872250 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:14:24.872228 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9\": container with ID starting with 626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9 not found: ID does not exist" containerID="626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9" Oct 30 20:14:24.872337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.872258 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9"} err="failed to get container status \"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9\": rpc error: code = NotFound desc = could not find container \"626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9\": container with ID starting with 626e7985df5d5e4c194a80d36a61201df0522912835b7918ca91cb9d4489a0a9 not found: ID does not exist" Oct 30 20:14:24.886745 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.886716 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:24.893902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.893875 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ced832-openshift-default-77bb9c5c67-wgcsx"] Oct 30 20:14:24.908175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:24.908148 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:25.792783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:25.792750 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:26.050647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:26.050554 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" path="/var/lib/kubelet/pods/5b18ac19-913d-42bc-bcad-a2ea9bc26439/volumes" Oct 30 20:14:26.681936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:26.680372 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:27.490326 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:27.490297 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:28.266497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:28.266294 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:29.074287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:29.074252 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:29.376794 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:14:29.376759 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:14:30.005585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:30.005553 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:30.422126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:30.422094 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:30.881646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:30.881615 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.908504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:31.908473 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.121899 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.121859 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:14:32.122509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122484 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" containerName="istio-proxy" Oct 30 20:14:32.122509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122508 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" containerName="istio-proxy" Oct 30 20:14:32.122833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122567 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" Oct 30 20:14:32.122833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122575 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" Oct 30 20:14:32.122833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122775 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db2cd8e-f002-40ae-9ca1-66041fdec5bd" containerName="istio-proxy" Oct 30 20:14:32.122833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.122801 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b18ac19-913d-42bc-bcad-a2ea9bc26439" containerName="istio-proxy" Oct 30 20:14:32.128000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.127978 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.143156 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.143131 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:14:32.160758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.160664 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.160758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.160749 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.160962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.160785 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.160962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.160816 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt28r\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.161127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.160982 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.161127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.161084 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.161242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.161125 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.161242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.161195 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.161242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.161223 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262014 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.261973 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262034 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262071 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262151 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262174 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262234 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262311 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262341 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262375 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt28r\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262456 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262551 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262787 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.262928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.262852 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.263095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.263050 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.264497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.264478 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.264848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.264825 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.283550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.283524 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt28r\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.287231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.287208 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.446193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.446094 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:32.734660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.734631 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:14:32.736675 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:32.736649 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0921becd_508b_4d8a_a854_b9ca1bff0633.slice/crio-a4f3c6f8e87341606ce14ac868e2849acd745eaf58fc5397a8d252bdc43750b3 WatchSource:0}: Error finding container a4f3c6f8e87341606ce14ac868e2849acd745eaf58fc5397a8d252bdc43750b3: Status 404 returned error can't find the container with id a4f3c6f8e87341606ce14ac868e2849acd745eaf58fc5397a8d252bdc43750b3 Oct 30 20:14:32.739344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.739304 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:32.739433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.739387 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:32.739477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.739429 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:32.909758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.909727 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:32.915557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.915516 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" event={"ID":"0921becd-508b-4d8a-a854-b9ca1bff0633","Type":"ContainerStarted","Data":"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472"} Oct 30 20:14:32.915728 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.915600 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" event={"ID":"0921becd-508b-4d8a-a854-b9ca1bff0633","Type":"ContainerStarted","Data":"a4f3c6f8e87341606ce14ac868e2849acd745eaf58fc5397a8d252bdc43750b3"} Oct 30 20:14:32.944489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:32.944428 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podStartSLOduration=0.944409286 podStartE2EDuration="944.409286ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:32.943118675 +0000 UTC m=+2633.010954244" watchObservedRunningTime="2026-10-30 20:14:32.944409286 +0000 UTC m=+2633.012244858" Oct 30 20:14:33.446793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:33.446751 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:33.448496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:33.448456 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:33.448685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:33.448528 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:34.323082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:34.319887 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:34.447094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:34.447035 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:34.447317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:34.447120 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:35.446655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:35.446571 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:35.447187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:35.446686 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:35.482896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:35.482865 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:36.434459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.434423 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n"] Oct 30 20:14:36.446494 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.446446 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:36.446689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.446508 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:36.451183 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.451153 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n"] Oct 30 20:14:36.451318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.451283 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.457020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.456997 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:36.457407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.457375 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:14:36.457544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.457444 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:14:36.457544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.457477 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:14:36.544398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.544355 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.544585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.544472 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrxgk\" (UniqueName: \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.544585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.544543 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.646226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.646190 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.646457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.646285 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.646457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.646318 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrxgk\" (UniqueName: \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.646938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.646910 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.648935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.648914 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.658731 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.658707 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrxgk\" (UniqueName: \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\") pod \"maas-api-revoke-sub-d41b534d-z764n\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:36.683027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.682995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:36.766155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:36.766045 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:37.087914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.087873 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n"] Oct 30 20:14:37.089206 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:37.089180 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e06b756_f108_4a93_9ffa_825963305da1.slice/crio-9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069 WatchSource:0}: Error finding container 9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069: Status 404 returned error can't find the container with id 9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069 Oct 30 20:14:37.447400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.447324 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:37.447400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.447361 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:37.720138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.716556 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:37.969180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.969144 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" event={"ID":"4e06b756-f108-4a93-9ffa-825963305da1","Type":"ContainerStarted","Data":"39dd1feac31e2b83ab034a54575e7df7f3f820de4b08434a286794b6f8cd030e"} Oct 30 20:14:37.969390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.969195 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" event={"ID":"4e06b756-f108-4a93-9ffa-825963305da1","Type":"ContainerStarted","Data":"9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069"} Oct 30 20:14:37.992539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:37.992447 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" podStartSLOduration=1.992433227 podStartE2EDuration="1.992433227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:37.987953999 +0000 UTC m=+2638.055789567" watchObservedRunningTime="2026-10-30 20:14:37.992433227 +0000 UTC m=+2638.060268795" Oct 30 20:14:38.446815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:38.446770 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" start-of-body= Oct 30 20:14:38.447106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:38.447052 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.200:15021/healthz/ready\": dial tcp 10.133.0.200:15021: connect: connection refused" Oct 30 20:14:38.908513 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:38.908479 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:38.981003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:38.980967 5733 generic.go:356] "Generic (PLEG): container finished" podID="4e06b756-f108-4a93-9ffa-825963305da1" containerID="39dd1feac31e2b83ab034a54575e7df7f3f820de4b08434a286794b6f8cd030e" exitCode=0 Oct 30 20:14:38.981223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:38.981050 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" event={"ID":"4e06b756-f108-4a93-9ffa-825963305da1","Type":"ContainerDied","Data":"39dd1feac31e2b83ab034a54575e7df7f3f820de4b08434a286794b6f8cd030e"} Oct 30 20:14:39.682690 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:39.682659 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:39.684990 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:39.683246 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:39.688447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:39.688411 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:14:39.914508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:39.914476 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:40.146458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.146428 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:40.187974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.187943 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\") pod \"4e06b756-f108-4a93-9ffa-825963305da1\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " Oct 30 20:14:40.188184 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.187999 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\" (UniqueName: \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\") pod \"4e06b756-f108-4a93-9ffa-825963305da1\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " Oct 30 20:14:40.188184 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.188178 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\") pod \"4e06b756-f108-4a93-9ffa-825963305da1\" (UID: \"4e06b756-f108-4a93-9ffa-825963305da1\") " Oct 30 20:14:40.188576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.188540 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca" pod "4e06b756-f108-4a93-9ffa-825963305da1" (UID: "4e06b756-f108-4a93-9ffa-825963305da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:40.190715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.190676 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca" pod "4e06b756-f108-4a93-9ffa-825963305da1" (UID: "4e06b756-f108-4a93-9ffa-825963305da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:40.190835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.190723 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk" pod "4e06b756-f108-4a93-9ffa-825963305da1" (UID: "4e06b756-f108-4a93-9ffa-825963305da1"). InnerVolumeSpecName "kube-api-access-xrxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:40.289612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.289535 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:40.289612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.289564 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrxgk\" (UniqueName: \"kubernetes.io/projected/4e06b756-f108-4a93-9ffa-825963305da1-kube-api-access-xrxgk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:40.289612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.289577 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e06b756-f108-4a93-9ffa-825963305da1-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:40.807487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.807452 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:40.995669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.995640 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" Oct 30 20:14:40.996090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.995675 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n" event={"ID":"4e06b756-f108-4a93-9ffa-825963305da1","Type":"ContainerDied","Data":"9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069"} Oct 30 20:14:40.996090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:40.995724 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d018912057ca5c1e02bde8c4a66268b098f53c837b996cdbe14bfbe6a055069" Oct 30 20:14:41.943877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:41.943841 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.711470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:42.711440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:43.498523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:43.498487 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:44.429521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:44.429482 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:45.220401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:45.220371 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:46.214276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.214240 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:46.659313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.659276 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv"] Oct 30 20:14:46.659743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.659722 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4e06b756-f108-4a93-9ffa-825963305da1" containerName="revoke-keys" Oct 30 20:14:46.659743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.659739 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e06b756-f108-4a93-9ffa-825963305da1" containerName="revoke-keys" Oct 30 20:14:46.659916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.659828 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e06b756-f108-4a93-9ffa-825963305da1" containerName="revoke-keys" Oct 30 20:14:46.662924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.662906 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.669153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.669128 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:14:46.669283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.669206 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:14:46.669567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.669547 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:14:46.669715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.669602 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:46.673565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.673541 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv"] Oct 30 20:14:46.752022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.751980 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.752211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.752041 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpvkt\" (UniqueName: \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.752259 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.752223 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.852816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.852784 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.852993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.852828 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpvkt\" (UniqueName: \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.852993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.852896 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.853582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.853557 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.855345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.855317 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.864906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.864881 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpvkt\" (UniqueName: \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\") pod \"maas-api-revoke-sub-fd586ec8-ck5gv\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:46.978520 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:46.978418 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:47.178661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.178623 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.201276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.201242 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:14:47.206733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.206707 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.230373 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.230305 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:14:47.257448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.257415 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.257678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.257658 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.257828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.257812 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258079 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258046 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258279 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258422 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258406 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptrcb\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258530 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258662 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.258782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.258769 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.280830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.280805 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv"] Oct 30 20:14:47.281597 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:47.281573 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f94acb_3082_4e59_b92f_32b4fa49eef4.slice/crio-c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0 WatchSource:0}: Error finding container c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0: Status 404 returned error can't find the container with id c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0 Oct 30 20:14:47.359796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359765 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359810 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359835 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptrcb\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359863 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359891 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359912 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.359957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.359945 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360104 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360145 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360213 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360253 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360666 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360507 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360703 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.360836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.360815 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.362726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.362704 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.362826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.362735 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.387026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.387003 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptrcb\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.394386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.394366 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\") pod \"e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.523499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.523427 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:47.842547 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:14:47.842500 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319ffc3d_d6be_43db_a083_6598ef046b75.slice/crio-9bf9416067f78ac215a5c7e6f5ed211667d3e91f1fb889d46b3db65b69486dc0 WatchSource:0}: Error finding container 9bf9416067f78ac215a5c7e6f5ed211667d3e91f1fb889d46b3db65b69486dc0: Status 404 returned error can't find the container with id 9bf9416067f78ac215a5c7e6f5ed211667d3e91f1fb889d46b3db65b69486dc0 Oct 30 20:14:47.845543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.845505 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:47.845650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.845579 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:47.845650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.845608 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:14:47.847503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:47.847477 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:14:48.052345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.052308 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" event={"ID":"319ffc3d-d6be-43db-a083-6598ef046b75","Type":"ContainerStarted","Data":"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d"} Oct 30 20:14:48.052546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.052363 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" event={"ID":"319ffc3d-d6be-43db-a083-6598ef046b75","Type":"ContainerStarted","Data":"9bf9416067f78ac215a5c7e6f5ed211667d3e91f1fb889d46b3db65b69486dc0"} Oct 30 20:14:48.052546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.052386 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" event={"ID":"65f94acb-3082-4e59-b92f-32b4fa49eef4","Type":"ContainerStarted","Data":"a0a586661deec210dbddeddd48593bfb8ff2da67ab118589b16fca91748fecb9"} Oct 30 20:14:48.052546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.052403 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" event={"ID":"65f94acb-3082-4e59-b92f-32b4fa49eef4","Type":"ContainerStarted","Data":"c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0"} Oct 30 20:14:48.096530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.096468 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podStartSLOduration=1.09644765 podStartE2EDuration="1.09644765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:48.09433377 +0000 UTC m=+2648.162169352" watchObservedRunningTime="2026-10-30 20:14:48.09644765 +0000 UTC m=+2648.164283219" Oct 30 20:14:48.122256 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.122113 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" podStartSLOduration=2.122097662 podStartE2EDuration="2.122097662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:48.120119767 +0000 UTC m=+2648.187955336" watchObservedRunningTime="2026-10-30 20:14:48.122097662 +0000 UTC m=+2648.189933229" Oct 30 20:14:48.320150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.320114 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.526791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.525439 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:48.532082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.531186 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:48.532082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:48.531248 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:49.059537 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:49.059495 5733 generic.go:356] "Generic (PLEG): container finished" podID="65f94acb-3082-4e59-b92f-32b4fa49eef4" containerID="a0a586661deec210dbddeddd48593bfb8ff2da67ab118589b16fca91748fecb9" exitCode=0 Oct 30 20:14:49.059820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:49.059581 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" event={"ID":"65f94acb-3082-4e59-b92f-32b4fa49eef4","Type":"ContainerDied","Data":"a0a586661deec210dbddeddd48593bfb8ff2da67ab118589b16fca91748fecb9"} Oct 30 20:14:49.282509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:49.282452 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.526082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:49.524491 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:49.526082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:49.524568 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:50.098108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.097233 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.266496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.266471 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:50.396415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.396379 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\") pod \"65f94acb-3082-4e59-b92f-32b4fa49eef4\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " Oct 30 20:14:50.396610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.396445 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\" (UniqueName: \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\") pod \"65f94acb-3082-4e59-b92f-32b4fa49eef4\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " Oct 30 20:14:50.396610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.396491 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\") pod \"65f94acb-3082-4e59-b92f-32b4fa49eef4\" (UID: \"65f94acb-3082-4e59-b92f-32b4fa49eef4\") " Oct 30 20:14:50.396785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.396761 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca" pod "65f94acb-3082-4e59-b92f-32b4fa49eef4" (UID: "65f94acb-3082-4e59-b92f-32b4fa49eef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:50.396910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.396890 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:50.398634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.398605 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca" pod "65f94acb-3082-4e59-b92f-32b4fa49eef4" (UID: "65f94acb-3082-4e59-b92f-32b4fa49eef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:50.398788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.398686 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt" pod "65f94acb-3082-4e59-b92f-32b4fa49eef4" (UID: "65f94acb-3082-4e59-b92f-32b4fa49eef4"). InnerVolumeSpecName "kube-api-access-qpvkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:50.497988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.497950 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpvkt\" (UniqueName: \"kubernetes.io/projected/65f94acb-3082-4e59-b92f-32b4fa49eef4-kube-api-access-qpvkt\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:50.497988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.497980 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65f94acb-3082-4e59-b92f-32b4fa49eef4-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:50.528909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.528446 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:50.528909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.528509 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:50.942103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:50.942040 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:51.075098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.075049 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" Oct 30 20:14:51.075098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.075083 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv" event={"ID":"65f94acb-3082-4e59-b92f-32b4fa49eef4","Type":"ContainerDied","Data":"c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0"} Oct 30 20:14:51.075325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.075128 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1da9229eb42e47d41ced0a3cf90454ad7010761b386200db4abb8ed8dc0b4b0" Oct 30 20:14:51.525087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.524401 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:51.525087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.524460 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:51.895517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:51.895368 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.525080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:52.524453 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:52.525080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:52.524522 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:52.783972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:52.783884 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.958689 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:52.958647 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.530247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:53.526429 5733 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body= Oct 30 20:14:53.530247 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:53.526492 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" Oct 30 20:14:53.834132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:53.833573 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.211336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:54.211300 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.527929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:54.527857 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:54.528323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:54.528302 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:54.528910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:54.528893 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:14:54.715399 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:54.715363 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:55.584697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:55.584645 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:56.404432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:56.404389 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.195945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:57.195907 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.998553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:57.998514 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:58.615270 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:58.615229 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:58.934388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:58.934297 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:59.376842 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:14:59.376800 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:14:59.996761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:14:59.996727 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.221461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.221421 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:00.222416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.222388 5733 cpu_manager.go:414] "containerMap: removing container" podUID="65f94acb-3082-4e59-b92f-32b4fa49eef4" containerName="revoke-keys" Oct 30 20:15:00.222587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.222571 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f94acb-3082-4e59-b92f-32b4fa49eef4" containerName="revoke-keys" Oct 30 20:15:00.249873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.249790 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f94acb-3082-4e59-b92f-32b4fa49eef4" containerName="revoke-keys" Oct 30 20:15:00.263083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.259290 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:00.266201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.266174 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.306081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.300246 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:00.306081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.300304 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:00.306081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.300413 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.340914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.332369 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:15:00.340914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.333007 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:00.340914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.333126 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:00.404082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.365319 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.453911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.452566 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llttk\" (UniqueName: \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.453911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.452631 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf7lb\" (UniqueName: \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.453911 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.453086 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.554583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.554539 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.554878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.554852 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.555025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.555007 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llttk\" (UniqueName: \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.555163 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.555144 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf7lb\" (UniqueName: \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.556435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.556411 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.557620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.557561 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.588116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.588033 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llttk\" (UniqueName: \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\") pod \"maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.589576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.589516 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf7lb\" (UniqueName: \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\") pod \"maas-api-key-cleanup-29889855-2xxdn\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:00.773484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.773370 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:00.826957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:00.826926 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:01.068325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.068292 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:01.069329 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:01.069295 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676f263c_e21b_4a53_b764_464a884df14e.slice/crio-5f7a17cb6dae449420fbad7de32da315360d36948873c693d3a8fdbd7241b1a7 WatchSource:0}: Error finding container 5f7a17cb6dae449420fbad7de32da315360d36948873c693d3a8fdbd7241b1a7: Status 404 returned error can't find the container with id 5f7a17cb6dae449420fbad7de32da315360d36948873c693d3a8fdbd7241b1a7 Oct 30 20:15:01.141001 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.140974 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:01.144095 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:01.143816 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32495d21_9434_48c9_a6d1_bb43345c84f0.slice/crio-33550d3c5f92deac82dd919988ccd88c8441690e6d45e7f7536af12242e2de52 WatchSource:0}: Error finding container 33550d3c5f92deac82dd919988ccd88c8441690e6d45e7f7536af12242e2de52: Status 404 returned error can't find the container with id 33550d3c5f92deac82dd919988ccd88c8441690e6d45e7f7536af12242e2de52 Oct 30 20:15:01.149607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.149571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" event={"ID":"676f263c-e21b-4a53-b764-464a884df14e","Type":"ContainerStarted","Data":"5f7a17cb6dae449420fbad7de32da315360d36948873c693d3a8fdbd7241b1a7"} Oct 30 20:15:01.392875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.392842 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:01.403246 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.403216 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:15:01.618688 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.618658 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:01.627370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.627341 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p"] Oct 30 20:15:01.631859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.631837 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.649117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.648451 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p"] Oct 30 20:15:01.649493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.649463 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:01.769457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.769417 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.769673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.769496 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsrlw\" (UniqueName: \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.769673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.769628 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.871017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.870938 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.871209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.871022 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsrlw\" (UniqueName: \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.871209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.871072 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.871771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.871751 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.873420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.873399 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.886109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.886083 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsrlw\" (UniqueName: \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:01.952808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:01.952762 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:02.158565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.158464 5733 generic.go:356] "Generic (PLEG): container finished" podID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerID="826e6bfe8f05167ca9a1c1518152f95b1c2f2cb82e6e6cc327f29683e687477e" exitCode=60 Oct 30 20:15:02.159027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.158554 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" event={"ID":"32495d21-9434-48c9-a6d1-bb43345c84f0","Type":"ContainerDied","Data":"826e6bfe8f05167ca9a1c1518152f95b1c2f2cb82e6e6cc327f29683e687477e"} Oct 30 20:15:02.159027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.158609 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" event={"ID":"32495d21-9434-48c9-a6d1-bb43345c84f0","Type":"ContainerStarted","Data":"33550d3c5f92deac82dd919988ccd88c8441690e6d45e7f7536af12242e2de52"} Oct 30 20:15:02.159027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.158817 5733 scope.go:122] "RemoveContainer" containerID="826e6bfe8f05167ca9a1c1518152f95b1c2f2cb82e6e6cc327f29683e687477e" Oct 30 20:15:02.160425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.160369 5733 generic.go:356] "Generic (PLEG): container finished" podID="676f263c-e21b-4a53-b764-464a884df14e" containerID="5ab12d25f7b89fb38077aebe8ef393a60c6feb3067ea3a70f80386270c25d7bc" exitCode=60 Oct 30 20:15:02.160505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.160452 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" event={"ID":"676f263c-e21b-4a53-b764-464a884df14e","Type":"ContainerDied","Data":"5ab12d25f7b89fb38077aebe8ef393a60c6feb3067ea3a70f80386270c25d7bc"} Oct 30 20:15:02.160668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.160651 5733 scope.go:122] "RemoveContainer" containerID="5ab12d25f7b89fb38077aebe8ef393a60c6feb3067ea3a70f80386270c25d7bc" Oct 30 20:15:02.277947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.277914 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p"] Oct 30 20:15:02.312980 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:02.312908 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72a6fd8_620c_4e72_8b24_fc0901c74526.slice/crio-772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808 WatchSource:0}: Error finding container 772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808: Status 404 returned error can't find the container with id 772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808 Oct 30 20:15:02.818405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:02.818322 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:03.169464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.169430 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" event={"ID":"d72a6fd8-620c-4e72-8b24-fc0901c74526","Type":"ContainerStarted","Data":"9e7e20d21800a27da71dade109d5d960f2918e86934e3046beeffc58e17c077c"} Oct 30 20:15:03.169981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.169487 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" event={"ID":"d72a6fd8-620c-4e72-8b24-fc0901c74526","Type":"ContainerStarted","Data":"772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808"} Oct 30 20:15:03.171664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.171634 5733 generic.go:356] "Generic (PLEG): container finished" podID="676f263c-e21b-4a53-b764-464a884df14e" containerID="471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6" exitCode=60 Oct 30 20:15:03.171772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.171712 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" event={"ID":"676f263c-e21b-4a53-b764-464a884df14e","Type":"ContainerDied","Data":"471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6"} Oct 30 20:15:03.171847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.171772 5733 scope.go:122] "RemoveContainer" containerID="5ab12d25f7b89fb38077aebe8ef393a60c6feb3067ea3a70f80386270c25d7bc" Oct 30 20:15:03.171957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.171939 5733 scope.go:122] "RemoveContainer" containerID="471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6" Oct 30 20:15:03.172241 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:03.172216 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5_odh-ai-gateway-infra(676f263c-e21b-4a53-b764-464a884df14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" podUID="676f263c-e21b-4a53-b764-464a884df14e" Oct 30 20:15:03.173972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.173952 5733 generic.go:356] "Generic (PLEG): container finished" podID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerID="431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95" exitCode=60 Oct 30 20:15:03.174122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.173978 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" event={"ID":"32495d21-9434-48c9-a6d1-bb43345c84f0","Type":"ContainerDied","Data":"431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95"} Oct 30 20:15:03.174312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.174293 5733 scope.go:122] "RemoveContainer" containerID="431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95" Oct 30 20:15:03.174554 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:03.174533 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2xxdn_odh-ai-gateway-infra(32495d21-9434-48c9-a6d1-bb43345c84f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" Oct 30 20:15:03.184501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.184480 5733 scope.go:122] "RemoveContainer" containerID="826e6bfe8f05167ca9a1c1518152f95b1c2f2cb82e6e6cc327f29683e687477e" Oct 30 20:15:03.190318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.190271 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" podStartSLOduration=2.190251947 podStartE2EDuration="2.190251947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.189113739 +0000 UTC m=+2663.256949298" watchObservedRunningTime="2026-10-30 20:15:03.190251947 +0000 UTC m=+2663.258087520" Oct 30 20:15:03.600710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:03.600676 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.182243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:04.182213 5733 scope.go:122] "RemoveContainer" containerID="431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95" Oct 30 20:15:04.182694 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:04.182498 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2xxdn_odh-ai-gateway-infra(32495d21-9434-48c9-a6d1-bb43345c84f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" Oct 30 20:15:04.183616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:04.183585 5733 generic.go:356] "Generic (PLEG): container finished" podID="d72a6fd8-620c-4e72-8b24-fc0901c74526" containerID="9e7e20d21800a27da71dade109d5d960f2918e86934e3046beeffc58e17c077c" exitCode=0 Oct 30 20:15:04.183725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:04.183665 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" event={"ID":"d72a6fd8-620c-4e72-8b24-fc0901c74526","Type":"ContainerDied","Data":"9e7e20d21800a27da71dade109d5d960f2918e86934e3046beeffc58e17c077c"} Oct 30 20:15:04.185560 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:04.185543 5733 scope.go:122] "RemoveContainer" containerID="471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6" Oct 30 20:15:04.185795 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:04.185777 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5_odh-ai-gateway-infra(676f263c-e21b-4a53-b764-464a884df14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" podUID="676f263c-e21b-4a53-b764-464a884df14e" Oct 30 20:15:04.667203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:04.667163 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:05.337606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.337579 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:05.512105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.511978 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\" (UniqueName: \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\") pod \"d72a6fd8-620c-4e72-8b24-fc0901c74526\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " Oct 30 20:15:05.512105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.512090 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\") pod \"d72a6fd8-620c-4e72-8b24-fc0901c74526\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " Oct 30 20:15:05.512346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.512197 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\") pod \"d72a6fd8-620c-4e72-8b24-fc0901c74526\" (UID: \"d72a6fd8-620c-4e72-8b24-fc0901c74526\") " Oct 30 20:15:05.512581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.512554 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca" pod "d72a6fd8-620c-4e72-8b24-fc0901c74526" (UID: "d72a6fd8-620c-4e72-8b24-fc0901c74526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:05.514335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.514307 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw" pod "d72a6fd8-620c-4e72-8b24-fc0901c74526" (UID: "d72a6fd8-620c-4e72-8b24-fc0901c74526"). InnerVolumeSpecName "kube-api-access-nsrlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:05.514335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.514330 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca" pod "d72a6fd8-620c-4e72-8b24-fc0901c74526" (UID: "d72a6fd8-620c-4e72-8b24-fc0901c74526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:05.613800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.613762 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsrlw\" (UniqueName: \"kubernetes.io/projected/d72a6fd8-620c-4e72-8b24-fc0901c74526-kube-api-access-nsrlw\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.613800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.613795 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.613800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.613806 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d72a6fd8-620c-4e72-8b24-fc0901c74526-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:05.761907 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:05.761872 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:06.199432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:06.199398 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" event={"ID":"d72a6fd8-620c-4e72-8b24-fc0901c74526","Type":"ContainerDied","Data":"772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808"} Oct 30 20:15:06.199432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:06.199433 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p" Oct 30 20:15:06.199619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:06.199449 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="772eafd19a6e68bbf62e754c5ab728ada2c061b062e5a3d8e73e40ee9a0d3808" Oct 30 20:15:06.696573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:06.696538 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.493817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:07.493783 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.308748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:08.308714 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:09.098790 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:09.098757 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:09.849687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:09.849616 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.686692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:10.686649 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.499491 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:11.499458 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.694515 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:11.694476 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 20:15:11.800208 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:11.800085 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.966846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:11.966807 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:12.489724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:12.489692 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.257409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:13.257368 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.416222 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:13.416187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.452165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:13.452039 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p"] Oct 30 20:15:13.457697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:13.457669 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-274922-j7s5p"] Oct 30 20:15:13.573786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:13.573687 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.050460 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:14.050429 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d72a6fd8-620c-4e72-8b24-fc0901c74526" path="/var/lib/kubelet/pods/d72a6fd8-620c-4e72-8b24-fc0901c74526/volumes" Oct 30 20:15:14.055509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:14.055484 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.506730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:14.506696 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.591336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:15.591289 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:16.426867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.426815 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerName="main" containerID="cri-o://406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99" gracePeriod=45 Oct 30 20:15:16.456956 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.456910 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.192:8000/ready\": dial tcp 10.133.0.192:8000: connect: connection refused" Oct 30 20:15:16.599409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.599372 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:16.694948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.694917 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" containerID="cri-o://01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2" gracePeriod=25 Oct 30 20:15:16.702021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.701997 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:15:16.747627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747579 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.747829 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747638 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\" (UniqueName: \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.747829 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747690 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.747829 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747770 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.748017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747847 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.748017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747888 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:16.748017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.747907 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\") pod \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\" (UID: \"fa02c1ea-cceb-4e21-a092-8abf6fa058ad\") " Oct 30 20:15:16.748240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.748130 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:16.748881 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.748845 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:16.749053 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.748943 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.749213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.749125 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.750828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.750794 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "kube-api-access-b4xfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:16.750974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.750934 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:16.751244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.751221 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm" pod "fa02c1ea-cceb-4e21-a092-8abf6fa058ad" (UID: "fa02c1ea-cceb-4e21-a092-8abf6fa058ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:16.849858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.849817 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.849858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.849850 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.850088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.849865 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xfz\" (UniqueName: \"kubernetes.io/projected/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-kube-api-access-b4xfz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:16.850088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:16.849879 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa02c1ea-cceb-4e21-a092-8abf6fa058ad-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:17.043109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.043006 5733 scope.go:122] "RemoveContainer" containerID="431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95" Oct 30 20:15:17.280941 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.280907 5733 generic.go:356] "Generic (PLEG): container finished" podID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerID="406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99" exitCode=0 Oct 30 20:15:17.281190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.280954 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" event={"ID":"fa02c1ea-cceb-4e21-a092-8abf6fa058ad","Type":"ContainerDied","Data":"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99"} Oct 30 20:15:17.281190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.280960 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" Oct 30 20:15:17.281190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.280996 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6" event={"ID":"fa02c1ea-cceb-4e21-a092-8abf6fa058ad","Type":"ContainerDied","Data":"48109a431058bc09ba6614507afea5af42b36bdec9be1da85c16e0f44cc7554e"} Oct 30 20:15:17.281190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.281015 5733 scope.go:122] "RemoveContainer" containerID="406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99" Oct 30 20:15:17.291800 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.291779 5733 scope.go:122] "RemoveContainer" containerID="406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99" Oct 30 20:15:17.292163 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:17.292035 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99\": container with ID starting with 406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99 not found: ID does not exist" containerID="406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99" Oct 30 20:15:17.292277 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.292183 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99"} err="failed to get container status \"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99\": rpc error: code = NotFound desc = could not find container \"406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99\": container with ID starting with 406bd9896ab30b79a3cb18bcb651c07009c8be9131fc13a5ec7bbd41fde19f99 not found: ID does not exist" Oct 30 20:15:17.306816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.306761 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:15:17.310193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.310165 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae2147/auth-test-model-ae2147-kserve-6888d97957-8nxc6"] Oct 30 20:15:17.432308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:17.432274 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:18.042552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.042518 5733 scope.go:122] "RemoveContainer" containerID="471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6" Oct 30 20:15:18.050877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.050851 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" path="/var/lib/kubelet/pods/fa02c1ea-cceb-4e21-a092-8abf6fa058ad/volumes" Oct 30 20:15:18.289085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.289037 5733 generic.go:356] "Generic (PLEG): container finished" podID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerID="8433acf49717d97556f8ffe02084e26d56c7e85049e0ac9b693f68f87a9af144" exitCode=60 Oct 30 20:15:18.289268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.289116 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" event={"ID":"32495d21-9434-48c9-a6d1-bb43345c84f0","Type":"ContainerDied","Data":"8433acf49717d97556f8ffe02084e26d56c7e85049e0ac9b693f68f87a9af144"} Oct 30 20:15:18.289268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.289180 5733 scope.go:122] "RemoveContainer" containerID="431add9e47a4d35a777637ff9f190779c63226b55f55dcca5ec68f237c371b95" Oct 30 20:15:18.289553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.289532 5733 scope.go:122] "RemoveContainer" containerID="8433acf49717d97556f8ffe02084e26d56c7e85049e0ac9b693f68f87a9af144" Oct 30 20:15:18.289873 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:18.289849 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889855-2xxdn_odh-ai-gateway-infra(32495d21-9434-48c9-a6d1-bb43345c84f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" Oct 30 20:15:18.378693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:18.378662 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.174404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.174367 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.307252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.307220 5733 generic.go:356] "Generic (PLEG): container finished" podID="676f263c-e21b-4a53-b764-464a884df14e" containerID="9bf587091d3c9778e5aeeb3692d015852d33ecbf9cc045bb52bd2624ba2bb6ea" exitCode=60 Oct 30 20:15:19.307252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.307255 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" event={"ID":"676f263c-e21b-4a53-b764-464a884df14e","Type":"ContainerDied","Data":"9bf587091d3c9778e5aeeb3692d015852d33ecbf9cc045bb52bd2624ba2bb6ea"} Oct 30 20:15:19.307544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.307358 5733 scope.go:122] "RemoveContainer" containerID="471b8fdb060bae111a5a198e79a427a45001dc0353892982eb3d895ac4d422e6" Oct 30 20:15:19.307735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.307713 5733 scope.go:122] "RemoveContainer" containerID="9bf587091d3c9778e5aeeb3692d015852d33ecbf9cc045bb52bd2624ba2bb6ea" Oct 30 20:15:19.308100 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:19.308041 5733 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5_odh-ai-gateway-infra(676f263c-e21b-4a53-b764-464a884df14e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" podUID="676f263c-e21b-4a53-b764-464a884df14e" Oct 30 20:15:19.340892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.340859 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:19.489974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.489948 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:19.580721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.580690 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\" (UniqueName: \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\") pod \"32495d21-9434-48c9-a6d1-bb43345c84f0\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " Oct 30 20:15:19.580896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.580798 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\") pod \"32495d21-9434-48c9-a6d1-bb43345c84f0\" (UID: \"32495d21-9434-48c9-a6d1-bb43345c84f0\") " Oct 30 20:15:19.581163 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.581139 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca" pod "32495d21-9434-48c9-a6d1-bb43345c84f0" (UID: "32495d21-9434-48c9-a6d1-bb43345c84f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.582913 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.582890 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb" pod "32495d21-9434-48c9-a6d1-bb43345c84f0" (UID: "32495d21-9434-48c9-a6d1-bb43345c84f0"). InnerVolumeSpecName "kube-api-access-kf7lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.682238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.682194 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf7lb\" (UniqueName: \"kubernetes.io/projected/32495d21-9434-48c9-a6d1-bb43345c84f0-kube-api-access-kf7lb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.682238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:19.682231 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32495d21-9434-48c9-a6d1-bb43345c84f0-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.213352 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.213304 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.319681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.319603 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" Oct 30 20:15:20.319958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.319609 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn" event={"ID":"32495d21-9434-48c9-a6d1-bb43345c84f0","Type":"ContainerDied","Data":"33550d3c5f92deac82dd919988ccd88c8441690e6d45e7f7536af12242e2de52"} Oct 30 20:15:20.319958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.319730 5733 scope.go:122] "RemoveContainer" containerID="8433acf49717d97556f8ffe02084e26d56c7e85049e0ac9b693f68f87a9af144" Oct 30 20:15:20.342126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.342095 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:20.352602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.352575 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889855-2xxdn"] Oct 30 20:15:20.371556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.371528 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:20.516458 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.516431 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:20.591356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.591271 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\") pod \"676f263c-e21b-4a53-b764-464a884df14e\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " Oct 30 20:15:20.591517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.591373 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\" (UniqueName: \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\") pod \"676f263c-e21b-4a53-b764-464a884df14e\" (UID: \"676f263c-e21b-4a53-b764-464a884df14e\") " Oct 30 20:15:20.591622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.591601 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca" pod "676f263c-e21b-4a53-b764-464a884df14e" (UID: "676f263c-e21b-4a53-b764-464a884df14e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:20.591710 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.591697 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/676f263c-e21b-4a53-b764-464a884df14e-maas-api-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.593609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.593586 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk" pod "676f263c-e21b-4a53-b764-464a884df14e" (UID: "676f263c-e21b-4a53-b764-464a884df14e"). InnerVolumeSpecName "kube-api-access-llttk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:20.692748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.692712 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llttk\" (UniqueName: \"kubernetes.io/projected/676f263c-e21b-4a53-b764-464a884df14e-kube-api-access-llttk\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.986475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:20.986425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:21.329528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.329446 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" Oct 30 20:15:21.329528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.329480 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5" event={"ID":"676f263c-e21b-4a53-b764-464a884df14e","Type":"ContainerDied","Data":"5f7a17cb6dae449420fbad7de32da315360d36948873c693d3a8fdbd7241b1a7"} Oct 30 20:15:21.330051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.329545 5733 scope.go:122] "RemoveContainer" containerID="9bf587091d3c9778e5aeeb3692d015852d33ecbf9cc045bb52bd2624ba2bb6ea" Oct 30 20:15:21.353123 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.353094 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:21.359968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.359939 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-8a8207-29889855-smqv5"] Oct 30 20:15:21.599661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.599627 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:21.879424 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:21.879337 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.051692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.051655 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" path="/var/lib/kubelet/pods/32495d21-9434-48c9-a6d1-bb43345c84f0/volumes" Oct 30 20:15:22.051692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.051683 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.052177 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.052159 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676f263c-e21b-4a53-b764-464a884df14e" path="/var/lib/kubelet/pods/676f263c-e21b-4a53-b764-464a884df14e/volumes" Oct 30 20:15:22.199456 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.199371 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.389205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.389174 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.712889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.712855 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:22.993667 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.993586 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk"] Oct 30 20:15:22.994037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994024 5733 cpu_manager.go:414] "containerMap: removing container" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994039 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994051 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d72a6fd8-620c-4e72-8b24-fc0901c74526" containerName="revoke-keys" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994068 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d72a6fd8-620c-4e72-8b24-fc0901c74526" containerName="revoke-keys" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994075 5733 cpu_manager.go:414] "containerMap: removing container" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994081 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994090 5733 cpu_manager.go:414] "containerMap: removing container" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994094 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994109 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994113 5733 cpu_manager.go:414] "containerMap: removing container" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerName="main" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994118 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerName="main" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994128 5733 cpu_manager.go:414] "containerMap: removing container" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994132 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994141 5733 cpu_manager.go:414] "containerMap: removing container" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994145 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994218 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994225 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d72a6fd8-620c-4e72-8b24-fc0901c74526" containerName="revoke-keys" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994232 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994237 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994245 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa02c1ea-cceb-4e21-a092-8abf6fa058ad" containerName="main" Oct 30 20:15:22.994394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:22.994251 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:23.009713 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.009681 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.013288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.013261 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:23.013487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.013280 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:23.013579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.013342 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:23.014279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.014256 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:15:23.014833 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.014810 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk"] Oct 30 20:15:23.122495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.122454 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvh9\" (UniqueName: \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.122679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.122533 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.122679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.122635 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.223276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.223234 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvh9\" (UniqueName: \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.223479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.223326 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.223479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.223389 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.224010 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.223981 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.225930 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.225906 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.235292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.235262 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvh9\" (UniqueName: \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.329402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.329314 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:23.621409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.621372 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk"] Oct 30 20:15:23.623325 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:23.623289 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb632b89_9893_4bec_99f8_3fddb1698816.slice/crio-9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a WatchSource:0}: Error finding container 9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a: Status 404 returned error can't find the container with id 9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a Oct 30 20:15:23.893872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:23.893788 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:24.359121 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:24.359077 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" event={"ID":"cb632b89-9893-4bec-99f8-3fddb1698816","Type":"ContainerStarted","Data":"c9cff19bc50619926bbb49d971aea6c4573bb36ed76b6990cc522fa17a90c208"} Oct 30 20:15:24.359299 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:24.359144 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" event={"ID":"cb632b89-9893-4bec-99f8-3fddb1698816","Type":"ContainerStarted","Data":"9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a"} Oct 30 20:15:24.379746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:24.379693 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" podStartSLOduration=2.379676796 podStartE2EDuration="2.379676796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:24.375996248 +0000 UTC m=+2684.443831817" watchObservedRunningTime="2026-10-30 20:15:24.379676796 +0000 UTC m=+2684.447512364" Oct 30 20:15:24.702012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:24.701978 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.367187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:25.367152 5733 generic.go:356] "Generic (PLEG): container finished" podID="cb632b89-9893-4bec-99f8-3fddb1698816" containerID="c9cff19bc50619926bbb49d971aea6c4573bb36ed76b6990cc522fa17a90c208" exitCode=0 Oct 30 20:15:25.367413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:25.367247 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" event={"ID":"cb632b89-9893-4bec-99f8-3fddb1698816","Type":"ContainerDied","Data":"c9cff19bc50619926bbb49d971aea6c4573bb36ed76b6990cc522fa17a90c208"} Oct 30 20:15:25.597038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:25.596984 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:26.499107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.499041 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:26.531234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.531212 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:26.660967 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.660891 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\") pod \"cb632b89-9893-4bec-99f8-3fddb1698816\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " Oct 30 20:15:26.661154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.660988 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\") pod \"cb632b89-9893-4bec-99f8-3fddb1698816\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " Oct 30 20:15:26.661154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.661031 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\" (UniqueName: \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\") pod \"cb632b89-9893-4bec-99f8-3fddb1698816\" (UID: \"cb632b89-9893-4bec-99f8-3fddb1698816\") " Oct 30 20:15:26.661354 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.661328 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca" pod "cb632b89-9893-4bec-99f8-3fddb1698816" (UID: "cb632b89-9893-4bec-99f8-3fddb1698816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:26.661434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.661404 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.663102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.663078 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9" pod "cb632b89-9893-4bec-99f8-3fddb1698816" (UID: "cb632b89-9893-4bec-99f8-3fddb1698816"). InnerVolumeSpecName "kube-api-access-rvvh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:26.663197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.663106 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca" pod "cb632b89-9893-4bec-99f8-3fddb1698816" (UID: "cb632b89-9893-4bec-99f8-3fddb1698816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:26.762777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.762740 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb632b89-9893-4bec-99f8-3fddb1698816-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:26.762777 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:26.762771 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvh9\" (UniqueName: \"kubernetes.io/projected/cb632b89-9893-4bec-99f8-3fddb1698816-kube-api-access-rvvh9\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:27.364697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:27.364666 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:27.381591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:27.381563 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" Oct 30 20:15:27.381730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:27.381603 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk" event={"ID":"cb632b89-9893-4bec-99f8-3fddb1698816","Type":"ContainerDied","Data":"9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a"} Oct 30 20:15:27.381730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:27.381649 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e270f454387e496b8ab50f8d20dc8c09148cafcd3851f24eebd08d9bab5037a" Oct 30 20:15:28.200266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:28.200232 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:29.004099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:29.004019 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:29.376672 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:29.376637 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:15:29.815831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:29.815755 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.719032 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:30.719000 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:31.600261 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:31.600219 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:15:31.717106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:31.717049 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.722140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:32.722095 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.154821 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.154789 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.370129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.367732 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.512585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.512494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.667673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.667641 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.802558 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.802472 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk"] Oct 30 20:15:33.806208 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.806179 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae2147-8q7bk"] Oct 30 20:15:33.868453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:33.868420 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:34.033929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:34.033894 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:34.049319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:34.049290 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb632b89-9893-4bec-99f8-3fddb1698816" path="/var/lib/kubelet/pods/cb632b89-9893-4bec-99f8-3fddb1698816/volumes" Oct 30 20:15:34.468711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:34.468677 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:35.027898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:35.027865 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:36.347908 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:36.347877 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:36.954618 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:36.954543 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 20:15:37.076714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.076680 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\" (UniqueName: \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\") pod \"66363c5c-5f40-47f5-bb62-4b00184dba74\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " Oct 30 20:15:37.076902 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.076728 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\") pod \"66363c5c-5f40-47f5-bb62-4b00184dba74\" (UID: \"66363c5c-5f40-47f5-bb62-4b00184dba74\") " Oct 30 20:15:37.077172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.077149 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume" pod "66363c5c-5f40-47f5-bb62-4b00184dba74" (UID: "66363c5c-5f40-47f5-bb62-4b00184dba74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:37.078837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.078817 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls" pod "66363c5c-5f40-47f5-bb62-4b00184dba74" (UID: "66363c5c-5f40-47f5-bb62-4b00184dba74"). InnerVolumeSpecName "kube-api-access-bcqls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:37.177614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.177581 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqls\" (UniqueName: \"kubernetes.io/projected/66363c5c-5f40-47f5-bb62-4b00184dba74-kube-api-access-bcqls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:37.177614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.177611 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66363c5c-5f40-47f5-bb62-4b00184dba74-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:37.455606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.455569 5733 generic.go:356] "Generic (PLEG): container finished" podID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerID="01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2" exitCode=0 Oct 30 20:15:37.456110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.455617 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" Oct 30 20:15:37.456110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.455662 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" event={"ID":"66363c5c-5f40-47f5-bb62-4b00184dba74","Type":"ContainerDied","Data":"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2"} Oct 30 20:15:37.456110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.455717 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v" event={"ID":"66363c5c-5f40-47f5-bb62-4b00184dba74","Type":"ContainerDied","Data":"2b3c6a65744dc01af3021a6563d2173242eae1b6cf1ea3b46d83aaa6da802b9c"} Oct 30 20:15:37.456110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.455740 5733 scope.go:122] "RemoveContainer" containerID="01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2" Oct 30 20:15:37.467106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.467087 5733 scope.go:122] "RemoveContainer" containerID="01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2" Oct 30 20:15:37.467389 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:37.467366 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2\": container with ID starting with 01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2 not found: ID does not exist" containerID="01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2" Oct 30 20:15:37.467448 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.467399 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2"} err="failed to get container status \"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2\": rpc error: code = NotFound desc = could not find container \"01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2\": container with ID starting with 01814610ade2667f642dbf44148688a6ad0321832a93074ad92aea13248132d2 not found: ID does not exist" Oct 30 20:15:37.483005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.482979 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 20:15:37.492767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.492744 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-274922-7b5cc55959-7st2v"] Oct 30 20:15:37.810610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:37.810510 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.057905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:38.057867 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" path="/var/lib/kubelet/pods/66363c5c-5f40-47f5-bb62-4b00184dba74/volumes" Oct 30 20:15:38.118132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:38.118089 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.927572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:38.927541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.868459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:39.868425 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.842199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:40.842126 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.873147 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:40.873119 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:15:40.881272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:40.881250 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:15:41.852672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:41.852633 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.213434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.213333 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.389701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.389670 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.472447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.472378 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:15:42.472627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.472604 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" containerID="cri-o://2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774" gracePeriod=30 Oct 30 20:15:42.483069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.483034 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:15:42.483288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.483268 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" containerID="cri-o://6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d" gracePeriod=30 Oct 30 20:15:42.509218 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.509190 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:15:42.509435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.509412 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" containerID="cri-o://8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472" gracePeriod=30 Oct 30 20:15:42.565918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.565885 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:15:42.566215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.566159 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" containerID="cri-o://8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3" gracePeriod=30 Oct 30 20:15:42.621120 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.621031 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.804599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:42.804521 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.109302 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.109248 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.267272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267233 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585"] Oct 30 20:15:43.267729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267713 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cb632b89-9893-4bec-99f8-3fddb1698816" containerName="revoke-keys" Oct 30 20:15:43.267813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267730 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb632b89-9893-4bec-99f8-3fddb1698816" containerName="revoke-keys" Oct 30 20:15:43.267813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267759 5733 cpu_manager.go:414] "containerMap: removing container" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" Oct 30 20:15:43.267813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267764 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" Oct 30 20:15:43.267813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267784 5733 cpu_manager.go:414] "containerMap: removing container" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:43.267813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267789 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:43.267982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267858 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="66363c5c-5f40-47f5-bb62-4b00184dba74" containerName="payload-pre-processing" Oct 30 20:15:43.267982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267870 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="32495d21-9434-48c9-a6d1-bb43345c84f0" containerName="cleanup" Oct 30 20:15:43.267982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267880 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb632b89-9893-4bec-99f8-3fddb1698816" containerName="revoke-keys" Oct 30 20:15:43.267982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.267888 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="676f263c-e21b-4a53-b764-464a884df14e" containerName="cleanup" Oct 30 20:15:43.271218 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.271201 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.282207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.282184 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:43.282348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.282291 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:43.286598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.286576 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585"] Oct 30 20:15:43.329012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.328976 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:43.329231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.329217 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:15:43.347152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.347123 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.347288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.347161 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.347381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.347283 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8l9\" (UniqueName: \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.448134 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.448016 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8l9\" (UniqueName: \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.448319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.448176 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.448319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.448203 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.448870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.448846 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.450619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.450591 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.460927 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.460896 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8l9\" (UniqueName: \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\") pod \"maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.584417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.584383 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:43.881000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:43.880956 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585"] Oct 30 20:15:43.882420 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:43.882391 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe06c151_d582_4b39_a049_fd6841abdd57.slice/crio-f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb WatchSource:0}: Error finding container f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb: Status 404 returned error can't find the container with id f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb Oct 30 20:15:44.015612 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:44.015577 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:44.509037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:44.508940 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" event={"ID":"be06c151-d582-4b39-a049-fd6841abdd57","Type":"ContainerStarted","Data":"529ee5ff6c40cd918bf1c91d7bcc57f8b0d916693e2307e0bed49b33dbd85b1e"} Oct 30 20:15:44.509037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:44.509006 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" event={"ID":"be06c151-d582-4b39-a049-fd6841abdd57","Type":"ContainerStarted","Data":"f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb"} Oct 30 20:15:44.533251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:44.533194 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" podStartSLOduration=1.533180233 podStartE2EDuration="1.533180233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:44.528954133 +0000 UTC m=+2704.596789701" watchObservedRunningTime="2026-10-30 20:15:44.533180233 +0000 UTC m=+2704.601015800" Oct 30 20:15:45.207378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:45.207349 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.517566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:45.517477 5733 generic.go:356] "Generic (PLEG): container finished" podID="be06c151-d582-4b39-a049-fd6841abdd57" containerID="529ee5ff6c40cd918bf1c91d7bcc57f8b0d916693e2307e0bed49b33dbd85b1e" exitCode=0 Oct 30 20:15:45.517566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:45.517548 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" event={"ID":"be06c151-d582-4b39-a049-fd6841abdd57","Type":"ContainerDied","Data":"529ee5ff6c40cd918bf1c91d7bcc57f8b0d916693e2307e0bed49b33dbd85b1e"} Oct 30 20:15:46.148554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.148511 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:46.668733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.668708 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:46.697156 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.697129 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\") pod \"be06c151-d582-4b39-a049-fd6841abdd57\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " Oct 30 20:15:46.697309 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.697253 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\") pod \"be06c151-d582-4b39-a049-fd6841abdd57\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " Oct 30 20:15:46.697379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.697323 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\" (UniqueName: \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\") pod \"be06c151-d582-4b39-a049-fd6841abdd57\" (UID: \"be06c151-d582-4b39-a049-fd6841abdd57\") " Oct 30 20:15:46.697553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.697507 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca" pod "be06c151-d582-4b39-a049-fd6841abdd57" (UID: "be06c151-d582-4b39-a049-fd6841abdd57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:46.697799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.697776 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.699553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.699530 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9" pod "be06c151-d582-4b39-a049-fd6841abdd57" (UID: "be06c151-d582-4b39-a049-fd6841abdd57"). InnerVolumeSpecName "kube-api-access-fr8l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:46.703543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.703510 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca" pod "be06c151-d582-4b39-a049-fd6841abdd57" (UID: "be06c151-d582-4b39-a049-fd6841abdd57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:46.798818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.798745 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be06c151-d582-4b39-a049-fd6841abdd57-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.798818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:46.798776 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8l9\" (UniqueName: \"kubernetes.io/projected/be06c151-d582-4b39-a049-fd6841abdd57-kube-api-access-fr8l9\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.214997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.214961 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.533532 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.533503 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" Oct 30 20:15:47.533671 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.533521 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585" event={"ID":"be06c151-d582-4b39-a049-fd6841abdd57","Type":"ContainerDied","Data":"f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb"} Oct 30 20:15:47.533671 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.533582 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f960285ef22d7439b98ab6a3242eaa02ed850e65f8fd840b8bb364a0bc1519cb" Oct 30 20:15:47.786294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.785498 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:15:47.808931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.808896 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.808990 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809164 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809197 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.809285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809235 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809285 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809269 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809320 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809353 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809435 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809456 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.809524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809467 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\") pod \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\" (UID: \"ed2cc347-b3e0-414b-b67a-d4444cfcc575\") " Oct 30 20:15:47.809732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809710 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.809924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.809889 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.810843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.810030 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.810843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.810069 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.810843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.810086 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.810843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.810101 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.810843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.810097 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:47.819757 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.819720 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "kube-api-access-bwsrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:47.821224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.821185 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:47.821692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.821665 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.821692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.821676 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo" pod "ed2cc347-b3e0-414b-b67a-d4444cfcc575" (UID: "ed2cc347-b3e0-414b-b67a-d4444cfcc575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:47.857357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.857326 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:15:47.913446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913399 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913526 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913581 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913636 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913662 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913686 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913711 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913729 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.913831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.913777 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\") pod \"319ffc3d-d6be-43db-a083-6598ef046b75\" (UID: \"319ffc3d-d6be-43db-a083-6598ef046b75\") " Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914077 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914098 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914110 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914120 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwsrr\" (UniqueName: \"kubernetes.io/projected/ed2cc347-b3e0-414b-b67a-d4444cfcc575-kube-api-access-bwsrr\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914129 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed2cc347-b3e0-414b-b67a-d4444cfcc575-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914422 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914435 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.914662 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.915158 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.915825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.915345 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.916570 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.916426 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "kube-api-access-ptrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:47.916823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.916798 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:47.918495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.918471 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:47.918808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.918762 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo" pod "319ffc3d-d6be-43db-a083-6598ef046b75" (UID: "319ffc3d-d6be-43db-a083-6598ef046b75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:47.934715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.934691 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:15:47.937913 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:47.937890 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:15:48.015413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015331 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015382 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015414 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015438 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015483 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015511 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015552 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015582 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015604 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015623 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015652 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015682 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015708 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.015775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015740 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015789 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\") pod \"0921becd-508b-4d8a-a854-b9ca1bff0633\" (UID: \"0921becd-508b-4d8a-a854-b9ca1bff0633\") " Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015817 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015832 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015874 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.015946 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\") pod \"cc883896-81be-4f04-8855-26253c0e7055\" (UID: \"cc883896-81be-4f04-8855-26253c0e7055\") " Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016011 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016323 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016415 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016437 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/319ffc3d-d6be-43db-a083-6598ef046b75-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016452 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016465 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.016478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016479 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016493 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016506 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/319ffc3d-d6be-43db-a083-6598ef046b75-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016520 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016534 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/319ffc3d-d6be-43db-a083-6598ef046b75-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016548 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptrcb\" (UniqueName: \"kubernetes.io/projected/319ffc3d-d6be-43db-a083-6598ef046b75-kube-api-access-ptrcb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016563 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc883896-81be-4f04-8855-26253c0e7055-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016914 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.016911 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017000 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017142 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017149 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017158 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017324 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.017942 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.017890 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.019410 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "kube-api-access-pqlwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.019603 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "kube-api-access-gt28r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.019782 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.020957 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.021403 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.021563 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.021659 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo" pod "0921becd-508b-4d8a-a854-b9ca1bff0633" (UID: "0921becd-508b-4d8a-a854-b9ca1bff0633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:48.022089 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.021884 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo" pod "cc883896-81be-4f04-8855-26253c0e7055" (UID: "cc883896-81be-4f04-8855-26253c0e7055"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:48.024111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.024048 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 20:15:48.117778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117747 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlwz\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-kube-api-access-pqlwz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117774 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117783 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117795 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt28r\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-kube-api-access-gt28r\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117804 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117814 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0921becd-508b-4d8a-a854-b9ca1bff0633-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117822 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117831 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117840 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117848 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117856 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117866 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117874 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc883896-81be-4f04-8855-26253c0e7055-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117883 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc883896-81be-4f04-8855-26253c0e7055-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117890 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc883896-81be-4f04-8855-26253c0e7055-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.117997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.117899 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0921becd-508b-4d8a-a854-b9ca1bff0633-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.198650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.198614 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.394122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.394083 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.540503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.540468 5733 generic.go:356] "Generic (PLEG): container finished" podID="319ffc3d-d6be-43db-a083-6598ef046b75" containerID="6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d" exitCode=0 Oct 30 20:15:48.540693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.540515 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" Oct 30 20:15:48.540693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.540559 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" event={"ID":"319ffc3d-d6be-43db-a083-6598ef046b75","Type":"ContainerDied","Data":"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d"} Oct 30 20:15:48.540693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.540609 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl" event={"ID":"319ffc3d-d6be-43db-a083-6598ef046b75","Type":"ContainerDied","Data":"9bf9416067f78ac215a5c7e6f5ed211667d3e91f1fb889d46b3db65b69486dc0"} Oct 30 20:15:48.540693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.540638 5733 scope.go:122] "RemoveContainer" containerID="6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d" Oct 30 20:15:48.542187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.542161 5733 generic.go:356] "Generic (PLEG): container finished" podID="cc883896-81be-4f04-8855-26253c0e7055" containerID="8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3" exitCode=0 Oct 30 20:15:48.542333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.542217 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" Oct 30 20:15:48.542333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.542247 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" event={"ID":"cc883896-81be-4f04-8855-26253c0e7055","Type":"ContainerDied","Data":"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3"} Oct 30 20:15:48.542333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.542294 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q" event={"ID":"cc883896-81be-4f04-8855-26253c0e7055","Type":"ContainerDied","Data":"07cbe324b62c793986a60a8f9b9d3fc9847cfa11a89af58fe418b550d2215fa7"} Oct 30 20:15:48.549419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.549395 5733 generic.go:356] "Generic (PLEG): container finished" podID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerID="2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774" exitCode=0 Oct 30 20:15:48.549535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.549420 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" event={"ID":"ed2cc347-b3e0-414b-b67a-d4444cfcc575","Type":"ContainerDied","Data":"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774"} Oct 30 20:15:48.549535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.549444 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" Oct 30 20:15:48.549535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.549464 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c" event={"ID":"ed2cc347-b3e0-414b-b67a-d4444cfcc575","Type":"ContainerDied","Data":"8c02d6570877302173fe8f7c4e05d4c365d19412a65eb20e27003653f0b5e103"} Oct 30 20:15:48.551408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551303 5733 generic.go:356] "Generic (PLEG): container finished" podID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerID="8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472" exitCode=0 Oct 30 20:15:48.551408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551368 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" event={"ID":"0921becd-508b-4d8a-a854-b9ca1bff0633","Type":"ContainerDied","Data":"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472"} Oct 30 20:15:48.551573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551408 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" event={"ID":"0921becd-508b-4d8a-a854-b9ca1bff0633","Type":"ContainerDied","Data":"a4f3c6f8e87341606ce14ac868e2849acd745eaf58fc5397a8d252bdc43750b3"} Oct 30 20:15:48.551573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551534 5733 scope.go:122] "RemoveContainer" containerID="6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d" Oct 30 20:15:48.551698 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551664 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx" Oct 30 20:15:48.551964 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:48.551936 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d\": container with ID starting with 6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d not found: ID does not exist" containerID="6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d" Oct 30 20:15:48.552098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551975 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d"} err="failed to get container status \"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d\": rpc error: code = NotFound desc = could not find container \"6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d\": container with ID starting with 6f3d9a04fbdef78a0a87c441fc164a953e66f7d7a81c65b9cb9e1dabb8cff57d not found: ID does not exist" Oct 30 20:15:48.552098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.551994 5733 scope.go:122] "RemoveContainer" containerID="8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3" Oct 30 20:15:48.564486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.564445 5733 scope.go:122] "RemoveContainer" containerID="8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3" Oct 30 20:15:48.564801 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:48.564780 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3\": container with ID starting with 8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3 not found: ID does not exist" containerID="8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3" Oct 30 20:15:48.564901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.564811 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3"} err="failed to get container status \"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3\": rpc error: code = NotFound desc = could not find container \"8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3\": container with ID starting with 8b9fcd8766dbfedf5297dd3e9c21a755b9a497d36c2a9c0f749ae0ac19b492b3 not found: ID does not exist" Oct 30 20:15:48.564901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.564834 5733 scope.go:122] "RemoveContainer" containerID="2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774" Oct 30 20:15:48.566508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.566485 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:15:48.569677 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.569654 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-s7bpl"] Oct 30 20:15:48.574249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.574216 5733 scope.go:122] "RemoveContainer" containerID="2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774" Oct 30 20:15:48.574586 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:48.574543 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774\": container with ID starting with 2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774 not found: ID does not exist" containerID="2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774" Oct 30 20:15:48.574711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.574586 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774"} err="failed to get container status \"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774\": rpc error: code = NotFound desc = could not find container \"2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774\": container with ID starting with 2130ffb6011fec85713ec6d894e06994ebd0df6d8b168eed9ef668202a0f1774 not found: ID does not exist" Oct 30 20:15:48.574711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.574611 5733 scope.go:122] "RemoveContainer" containerID="8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472" Oct 30 20:15:48.577744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.577721 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:15:48.582209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.582191 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-x2s6q"] Oct 30 20:15:48.585865 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.585846 5733 scope.go:122] "RemoveContainer" containerID="8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472" Oct 30 20:15:48.586204 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:48.586185 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472\": container with ID starting with 8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472 not found: ID does not exist" containerID="8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472" Oct 30 20:15:48.586292 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.586211 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472"} err="failed to get container status \"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472\": rpc error: code = NotFound desc = could not find container \"8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472\": container with ID starting with 8e7bb8c8d4d7b173ac437a1d21f1bf6e7512e7ad02947d0a700975cdde4b5472 not found: ID does not exist" Oct 30 20:15:48.601371 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.601339 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:15:48.604362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.604341 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-h8n5c"] Oct 30 20:15:48.615868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.615841 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:15:48.622150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:48.622131 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae2147-openshift-default-7fc4875b99-jkfkx"] Oct 30 20:15:49.184382 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:49.184352 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.049174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:50.049141 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" path="/var/lib/kubelet/pods/0921becd-508b-4d8a-a854-b9ca1bff0633/volumes" Oct 30 20:15:50.049630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:50.049616 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" path="/var/lib/kubelet/pods/319ffc3d-d6be-43db-a083-6598ef046b75/volumes" Oct 30 20:15:50.050042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:50.050030 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc883896-81be-4f04-8855-26253c0e7055" path="/var/lib/kubelet/pods/cc883896-81be-4f04-8855-26253c0e7055/volumes" Oct 30 20:15:50.050767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:50.050753 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" path="/var/lib/kubelet/pods/ed2cc347-b3e0-414b-b67a-d4444cfcc575/volumes" Oct 30 20:15:50.158182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:50.158151 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:51.026461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:51.026378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:51.901319 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:51.901285 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.829282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:52.829249 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.993198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:52.993164 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.422496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:53.422468 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.562719 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:53.562688 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.702745 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:53.702652 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.182552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.182504 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.325205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325159 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb"] Oct 30 20:15:54.325720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325692 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" Oct 30 20:15:54.325720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325715 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" Oct 30 20:15:54.325720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325726 5733 cpu_manager.go:414] "containerMap: removing container" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325732 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325742 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325747 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325781 5733 cpu_manager.go:414] "containerMap: removing container" podUID="be06c151-d582-4b39-a049-fd6841abdd57" containerName="revoke-keys" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325785 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be06c151-d582-4b39-a049-fd6841abdd57" containerName="revoke-keys" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325801 5733 cpu_manager.go:414] "containerMap: removing container" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325806 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325896 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2cc347-b3e0-414b-b67a-d4444cfcc575" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325904 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="319ffc3d-d6be-43db-a083-6598ef046b75" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325911 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc883896-81be-4f04-8855-26253c0e7055" containerName="istio-proxy" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325918 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="be06c151-d582-4b39-a049-fd6841abdd57" containerName="revoke-keys" Oct 30 20:15:54.326005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.325926 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="0921becd-508b-4d8a-a854-b9ca1bff0633" containerName="istio-proxy" Oct 30 20:15:54.331116 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.331099 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.339012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.338989 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:15:54.341709 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.341682 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb"] Oct 30 20:15:54.355283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.355264 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:15:54.355837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.355812 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:54.355952 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.355842 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:15:54.383213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.383180 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhww\" (UniqueName: \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.383384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.383222 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.383384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.383378 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.484692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.484612 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.484692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.484673 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhww\" (UniqueName: \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.484945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.484698 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.485384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.485363 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.486961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.486938 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.509627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.509600 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhww\" (UniqueName: \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\") pod \"maas-api-revoke-sub-aea564a8-5jtnb\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.647849 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.647815 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:54.932629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:54.932603 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb"] Oct 30 20:15:54.934927 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:15:54.934898 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6891ecf_957f_4c8d_a833_ffc26ddc6789.slice/crio-5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4 WatchSource:0}: Error finding container 5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4: Status 404 returned error can't find the container with id 5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4 Oct 30 20:15:55.603598 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:55.603553 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" event={"ID":"e6891ecf-957f-4c8d-a833-ffc26ddc6789","Type":"ContainerStarted","Data":"e66c3f3e7400c2bf7b2343d3e99171cb242bc41f2eea0e36260d8612a658153c"} Oct 30 20:15:55.604017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:55.603653 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" event={"ID":"e6891ecf-957f-4c8d-a833-ffc26ddc6789","Type":"ContainerStarted","Data":"5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4"} Oct 30 20:15:55.621877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:55.621817 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" podStartSLOduration=1.621798556 podStartE2EDuration="1.621798556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:55.621439045 +0000 UTC m=+2715.689274614" watchObservedRunningTime="2026-10-30 20:15:55.621798556 +0000 UTC m=+2715.689634124" Oct 30 20:15:56.611207 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:56.611161 5733 generic.go:356] "Generic (PLEG): container finished" podID="e6891ecf-957f-4c8d-a833-ffc26ddc6789" containerID="e66c3f3e7400c2bf7b2343d3e99171cb242bc41f2eea0e36260d8612a658153c" exitCode=0 Oct 30 20:15:56.611625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:56.611227 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" event={"ID":"e6891ecf-957f-4c8d-a833-ffc26ddc6789","Type":"ContainerDied","Data":"e66c3f3e7400c2bf7b2343d3e99171cb242bc41f2eea0e36260d8612a658153c"} Oct 30 20:15:57.767470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.767439 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:57.824701 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.824665 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\") pod \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " Oct 30 20:15:57.824884 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.824747 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\" (UniqueName: \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\") pod \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " Oct 30 20:15:57.824976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.824880 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\") pod \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\" (UID: \"e6891ecf-957f-4c8d-a833-ffc26ddc6789\") " Oct 30 20:15:57.825140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.825116 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca" pod "e6891ecf-957f-4c8d-a833-ffc26ddc6789" (UID: "e6891ecf-957f-4c8d-a833-ffc26ddc6789"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:57.825331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.825312 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:57.827684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.827649 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww" pod "e6891ecf-957f-4c8d-a833-ffc26ddc6789" (UID: "e6891ecf-957f-4c8d-a833-ffc26ddc6789"). InnerVolumeSpecName "kube-api-access-lzhww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:57.828309 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.828283 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca" pod "e6891ecf-957f-4c8d-a833-ffc26ddc6789" (UID: "e6891ecf-957f-4c8d-a833-ffc26ddc6789"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:57.926355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.926263 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzhww\" (UniqueName: \"kubernetes.io/projected/e6891ecf-957f-4c8d-a833-ffc26ddc6789-kube-api-access-lzhww\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:57.926355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:57.926293 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6891ecf-957f-4c8d-a833-ffc26ddc6789-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:58.626892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:58.626854 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" event={"ID":"e6891ecf-957f-4c8d-a833-ffc26ddc6789","Type":"ContainerDied","Data":"5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4"} Oct 30 20:15:58.626892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:58.626877 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb" Oct 30 20:15:58.627139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:15:58.626913 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e66a6163ab9f05bd2d084253812a0a5b8f7e190138c14b10a202e255b1ba5c4" Oct 30 20:15:59.377689 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:15:59.377649 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:16:03.040649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.040608 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" containerID="cri-o://529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7" gracePeriod=45 Oct 30 20:16:03.044161 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.044120 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" containerID="cri-o://045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d" gracePeriod=45 Oct 30 20:16:03.300375 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.300311 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:16:03.325340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.325315 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:16:03.381336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381296 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381356 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.381501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381394 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\" (UniqueName: \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.381501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381462 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\" (UniqueName: \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.381626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381518 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\" (UniqueName: \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381548 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381583 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.381730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381624 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381664 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.381796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381755 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381832 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381796 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\") pod \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\" (UID: \"f3c61292-cd14-4230-80ac-2aaf2d55be7c\") " Oct 30 20:16:03.381867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.381831 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\") pod \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\" (UID: \"11fec734-033b-44bb-a7a4-5a3f81ecddeb\") " Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382409 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382428 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382456 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382734 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382776 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.383088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.382974 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.385301 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.385264 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:03.385827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.385765 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:03.386369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.386342 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "kube-api-access-qn666". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.386887 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.386851 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.387449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.387429 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm" pod "f3c61292-cd14-4230-80ac-2aaf2d55be7c" (UID: "f3c61292-cd14-4230-80ac-2aaf2d55be7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:03.387535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.387472 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl" pod "11fec734-033b-44bb-a7a4-5a3f81ecddeb" (UID: "11fec734-033b-44bb-a7a4-5a3f81ecddeb"). InnerVolumeSpecName "kube-api-access-j67sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:03.483360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483327 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483357 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn666\" (UniqueName: \"kubernetes.io/projected/f3c61292-cd14-4230-80ac-2aaf2d55be7c-kube-api-access-qn666\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483367 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483376 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483385 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483395 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483403 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483410 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483420 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483428 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3c61292-cd14-4230-80ac-2aaf2d55be7c-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483435 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11fec734-033b-44bb-a7a4-5a3f81ecddeb-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.483596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.483443 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67sl\" (UniqueName: \"kubernetes.io/projected/11fec734-033b-44bb-a7a4-5a3f81ecddeb-kube-api-access-j67sl\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:03.667004 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.666967 5733 generic.go:356] "Generic (PLEG): container finished" podID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerID="045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d" exitCode=0 Oct 30 20:16:03.667228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.667017 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" Oct 30 20:16:03.667228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.667077 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" event={"ID":"11fec734-033b-44bb-a7a4-5a3f81ecddeb","Type":"ContainerDied","Data":"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d"} Oct 30 20:16:03.667228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.667133 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97" event={"ID":"11fec734-033b-44bb-a7a4-5a3f81ecddeb","Type":"ContainerDied","Data":"fb69c8a5ea37d0cc3bc625279a43292caf9e49a46c15971ddffc03706779fdee"} Oct 30 20:16:03.667228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.667151 5733 scope.go:122] "RemoveContainer" containerID="045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d" Oct 30 20:16:03.668498 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.668473 5733 generic.go:356] "Generic (PLEG): container finished" podID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerID="529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7" exitCode=0 Oct 30 20:16:03.668616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.668524 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" Oct 30 20:16:03.668616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.668546 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" event={"ID":"f3c61292-cd14-4230-80ac-2aaf2d55be7c","Type":"ContainerDied","Data":"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7"} Oct 30 20:16:03.668616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.668571 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8" event={"ID":"f3c61292-cd14-4230-80ac-2aaf2d55be7c","Type":"ContainerDied","Data":"b4ca4f8964c48bcd24d93c0d7575361747264f0db0da49f23da72808a03f55c3"} Oct 30 20:16:03.679442 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.679425 5733 scope.go:122] "RemoveContainer" containerID="045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d" Oct 30 20:16:03.679748 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:16:03.679720 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d\": container with ID starting with 045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d not found: ID does not exist" containerID="045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d" Oct 30 20:16:03.679828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.679749 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d"} err="failed to get container status \"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d\": rpc error: code = NotFound desc = could not find container \"045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d\": container with ID starting with 045074a9c934b9e224efb309d647d509debbc6d3c865acde65ef2cb2d2c8628d not found: ID does not exist" Oct 30 20:16:03.679828 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.679766 5733 scope.go:122] "RemoveContainer" containerID="529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7" Oct 30 20:16:03.690162 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.690140 5733 scope.go:122] "RemoveContainer" containerID="529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7" Oct 30 20:16:03.690456 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:16:03.690436 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7\": container with ID starting with 529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7 not found: ID does not exist" containerID="529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7" Oct 30 20:16:03.690533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.690463 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7"} err="failed to get container status \"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7\": rpc error: code = NotFound desc = could not find container \"529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7\": container with ID starting with 529a6edf377688fb1c641417b062b61bf64560529457a39f2c134d508405a9c7 not found: ID does not exist" Oct 30 20:16:03.696145 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.696120 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 20:16:03.700894 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.700874 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-w3-8a8207-kserve-68f4987c5d-zbc97"] Oct 30 20:16:03.713653 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.713626 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 20:16:03.718265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:03.718242 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8a8207/distinct-2-w3-8a8207-kserve-8557c84f8f-r5kh8"] Oct 30 20:16:04.033606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:04.033524 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf"] Oct 30 20:16:04.037994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:04.037964 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd32a60d-98nkf"] Oct 30 20:16:04.049284 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:04.049255 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" path="/var/lib/kubelet/pods/11fec734-033b-44bb-a7a4-5a3f81ecddeb/volumes" Oct 30 20:16:04.049699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:04.049680 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac951dc0-28f6-4afd-a450-ea2adc788134" path="/var/lib/kubelet/pods/ac951dc0-28f6-4afd-a450-ea2adc788134/volumes" Oct 30 20:16:04.050112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:04.050093 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" path="/var/lib/kubelet/pods/f3c61292-cd14-4230-80ac-2aaf2d55be7c/volumes" Oct 30 20:16:09.508348 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:09.508287 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:09.692994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:09.692957 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:09.711007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:09.710972 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585"] Oct 30 20:16:09.716376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:09.716349 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8a8207-gs585"] Oct 30 20:16:09.846453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:09.846420 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:10.049157 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:10.049118 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be06c151-d582-4b39-a049-fd6841abdd57" path="/var/lib/kubelet/pods/be06c151-d582-4b39-a049-fd6841abdd57/volumes" Oct 30 20:16:10.308400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:10.308300 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.127477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:18.127441 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.332708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:18.332671 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.543451 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:18.543360 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:25.551830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:25.551679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:25.723028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:25.722990 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:25.930435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:25.930399 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.139742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:26.139705 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:27.937445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:27.937400 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:29.377131 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:16:29.377087 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:16:29.687051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:29.686962 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:29.849959 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:29.849924 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:33.772295 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:33.772260 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.697085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697032 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:16:47.697523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697475 5733 cpu_manager.go:414] "containerMap: removing container" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" Oct 30 20:16:47.697523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697487 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" Oct 30 20:16:47.697523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697519 5733 cpu_manager.go:414] "containerMap: removing container" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" Oct 30 20:16:47.697523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697524 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" Oct 30 20:16:47.697678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697533 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e6891ecf-957f-4c8d-a833-ffc26ddc6789" containerName="revoke-keys" Oct 30 20:16:47.697678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697538 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6891ecf-957f-4c8d-a833-ffc26ddc6789" containerName="revoke-keys" Oct 30 20:16:47.697678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697598 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c61292-cd14-4230-80ac-2aaf2d55be7c" containerName="main" Oct 30 20:16:47.697678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697607 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6891ecf-957f-4c8d-a833-ffc26ddc6789" containerName="revoke-keys" Oct 30 20:16:47.697678 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.697615 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fec734-033b-44bb-a7a4-5a3f81ecddeb" containerName="main" Oct 30 20:16:47.700787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.700770 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.721991 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.721956 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:16:47.745220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.745187 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:16:47.748818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.748796 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.775052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.775026 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:16:47.816763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.816733 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.816906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.816772 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.816906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.816853 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.816988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.816943 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4gj\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.817025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.817003 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.817093 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.817084 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.817134 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.817114 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.817174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.817132 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.817174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.817150 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918009 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.917981 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918019 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918038 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918075 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918142 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918228 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918265 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918290 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918319 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918457 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918496 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ks57\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918530 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918555 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918563 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918591 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918742 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.918820 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918778 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918831 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4gj\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918878 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918902 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918920 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.918993 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.919251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.919225 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.920605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.920584 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.920889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.920872 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.931085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.931039 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:47.931085 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:47.931052 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4gj\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\") pod \"maas-default-gateway-openshift-default-57f78bcd55-z9kxf\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:48.016451 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.016377 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:48.019748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.019715 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.019880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.019754 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.019955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.019889 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.019955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.019933 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.019974 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020024 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020097 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020082 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ks57\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020122 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020124 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020156 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020182 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020304 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020795 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.020896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.020838 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.022686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.022664 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.022798 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.022778 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.032522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.032497 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ks57\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.036829 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.036808 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpgwn\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.065370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.065333 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:48.307847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.307820 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:16:48.309215 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:16:48.309190 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eea678_8a13_45be_8ddd_010bf437f7dd.slice/crio-bb5984c29da714dafa54595216b3d0656d95086d73c9ee9397292c47ba0b4dee WatchSource:0}: Error finding container bb5984c29da714dafa54595216b3d0656d95086d73c9ee9397292c47ba0b4dee: Status 404 returned error can't find the container with id bb5984c29da714dafa54595216b3d0656d95086d73c9ee9397292c47ba0b4dee Oct 30 20:16:48.311778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.311745 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.311860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.311808 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.311860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.311837 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.367840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.367814 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:16:48.369512 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:16:48.369477 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532bb3e4_bc28_438c_b4ae_5c8590d62be1.slice/crio-8c164e7c96b80d6438911d72cc66a148c547d3f6d891ecbde0c88aad8b6a1d50 WatchSource:0}: Error finding container 8c164e7c96b80d6438911d72cc66a148c547d3f6d891ecbde0c88aad8b6a1d50: Status 404 returned error can't find the container with id 8c164e7c96b80d6438911d72cc66a148c547d3f6d891ecbde0c88aad8b6a1d50 Oct 30 20:16:48.371784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.371754 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.371868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.371819 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.371868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.371851 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:16:48.972155 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.972115 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" event={"ID":"a6eea678-8a13-45be-8ddd-010bf437f7dd","Type":"ContainerStarted","Data":"9e9762e2cffd76d54afd0d88e8d52cf74b185a0081707eea14f63288975d3c65"} Oct 30 20:16:48.972578 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.972184 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" event={"ID":"a6eea678-8a13-45be-8ddd-010bf437f7dd","Type":"ContainerStarted","Data":"bb5984c29da714dafa54595216b3d0656d95086d73c9ee9397292c47ba0b4dee"} Oct 30 20:16:48.973718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.973696 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" event={"ID":"532bb3e4-bc28-438c-b4ae-5c8590d62be1","Type":"ContainerStarted","Data":"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537"} Oct 30 20:16:48.973885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.973870 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" event={"ID":"532bb3e4-bc28-438c-b4ae-5c8590d62be1","Type":"ContainerStarted","Data":"8c164e7c96b80d6438911d72cc66a148c547d3f6d891ecbde0c88aad8b6a1d50"} Oct 30 20:16:48.997372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:48.997328 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podStartSLOduration=1.997315179 podStartE2EDuration="1.997315179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:48.993114345 +0000 UTC m=+2769.060949911" watchObservedRunningTime="2026-10-30 20:16:48.997315179 +0000 UTC m=+2769.065150746" Oct 30 20:16:49.031081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.027684 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:49.055082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.040493 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podStartSLOduration=2.040467467 podStartE2EDuration="2.040467467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.027391612 +0000 UTC m=+2769.095227181" watchObservedRunningTime="2026-10-30 20:16:49.040467467 +0000 UTC m=+2769.108303040" Oct 30 20:16:49.055082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.041184 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:49.055082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.041236 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:49.067385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.067341 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:49.068351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.068327 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:49.068433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:49.068371 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:50.026082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:50.022475 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:50.026082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:50.022535 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:50.066708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:50.066665 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:50.066880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:50.066724 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:51.027111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:51.023516 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:51.027111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:51.023577 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:51.066028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:51.065989 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:51.066235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:51.066043 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:52.019479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:52.019440 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:52.019740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:52.019710 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:52.066468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:52.066436 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:52.066864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:52.066483 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:52.931082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:52.927381 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.019082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:53.018447 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:53.019082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:53.018513 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:53.066666 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:53.066623 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:53.067189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:53.066686 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:54.018472 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.018433 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:54.018883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.018853 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:54.065711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.065672 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:54.065935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.065916 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:54.256403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.256365 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.476395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:54.476359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:55.017495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:55.017456 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 20:16:55.017683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:55.017521 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 20:16:55.067720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:55.067685 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body= Oct 30 20:16:55.067900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:55.067742 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" Oct 30 20:16:56.637948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.637894 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:56.638459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.638359 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:56.639646 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.639623 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:16:56.710570 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.710536 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:56.711405 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.711382 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:56.712334 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.712308 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:16:56.880140 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:16:56.880106 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:59.376450 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:16:59.376405 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:17:02.308797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:02.308762 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.110845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:11.110809 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.248504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:11.248470 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:22.885608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.885525 5733 scope.go:122] "RemoveContainer" containerID="01ba6c85150d4e792b44b173c615c9c4679bd1c6cc31132a1676eded0b87bb74" Oct 30 20:17:22.895381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.895353 5733 scope.go:122] "RemoveContainer" containerID="8bca8c7172b58617db212621da6fe28ee99b9b5c7f628cf0e8c1c27660a69fff" Oct 30 20:17:22.904897 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.904879 5733 scope.go:122] "RemoveContainer" containerID="996aec9e2c094afa81459ff3e81bf6e67b8c9ed1514ba598e728654841df7715" Oct 30 20:17:22.913461 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.913437 5733 scope.go:122] "RemoveContainer" containerID="d022ea1dd3925036e56c9cfa705c0f27539e170ecf434665b030bb36553e6564" Oct 30 20:17:22.922321 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.922300 5733 scope.go:122] "RemoveContainer" containerID="72056619d34752340c8c952489caf9900bc7c79d2467d9b793fb98320192ba41" Oct 30 20:17:22.931087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:22.931045 5733 scope.go:122] "RemoveContainer" containerID="a86f772ed299cd967a2b343596a6469607867289cf54ca127e247f90f1f0425d" Oct 30 20:17:29.376719 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:17:29.376678 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:17:32.969266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:32.969227 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 20:17:32.969810 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:32.969508 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" containerID="cri-o://1e2a47b0561d9eea882ec56a468c8499de6b30d12716a739608e4ad974587968" gracePeriod=30 Oct 30 20:17:33.258928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.258835 5733 generic.go:356] "Generic (PLEG): container finished" podID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerID="1e2a47b0561d9eea882ec56a468c8499de6b30d12716a739608e4ad974587968" exitCode=0 Oct 30 20:17:33.259133 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.258921 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" event={"ID":"9a7e5628-1d20-497b-a673-d28a54b125f2","Type":"ContainerDied","Data":"1e2a47b0561d9eea882ec56a468c8499de6b30d12716a739608e4ad974587968"} Oct 30 20:17:33.285163 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.285135 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 20:17:33.405559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405522 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.405559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405564 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.405839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405623 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.405839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405761 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.405975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405844 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.405975 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405920 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.406175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.405991 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\") pod \"9a7e5628-1d20-497b-a673-d28a54b125f2\" (UID: \"9a7e5628-1d20-497b-a673-d28a54b125f2\") " Oct 30 20:17:33.406705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.406663 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:33.408795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.408767 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:33.409275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.409247 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "kube-api-access-bsccg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:33.410337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.410287 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:33.410337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.410298 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:33.410337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.410310 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:33.410584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.410315 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token" pod "9a7e5628-1d20-497b-a673-d28a54b125f2" (UID: "9a7e5628-1d20-497b-a673-d28a54b125f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:33.507580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507541 5733 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-cacerts\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507583 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507601 5733 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-kubeconfig\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507616 5733 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7e5628-1d20-497b-a673-d28a54b125f2-local-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507631 5733 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-dns-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507645 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsccg\" (UniqueName: \"kubernetes.io/projected/9a7e5628-1d20-497b-a673-d28a54b125f2-kube-api-access-bsccg\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:33.507779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:33.507663 5733 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9a7e5628-1d20-497b-a673-d28a54b125f2-istio-csr-ca-configmap\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:34.286864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:34.285655 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" event={"ID":"9a7e5628-1d20-497b-a673-d28a54b125f2","Type":"ContainerDied","Data":"bbb1dcfa4a9faccd0c08cceb5f23329831e3497ec437246a202784668c493c2c"} Oct 30 20:17:34.286864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:34.285731 5733 scope.go:122] "RemoveContainer" containerID="1e2a47b0561d9eea882ec56a468c8499de6b30d12716a739608e4ad974587968" Oct 30 20:17:34.286864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:34.285891 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z" Oct 30 20:17:34.317449 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:34.317425 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 20:17:34.326282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:34.326260 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6h66z"] Oct 30 20:17:36.035702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:36.035658 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz"] Oct 30 20:17:36.038649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:36.038627 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b40ceac1-6b8tz"] Oct 30 20:17:36.049342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:36.049316 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a7e7bd8-88c7-465c-be3d-fce254debc65" path="/var/lib/kubelet/pods/5a7e7bd8-88c7-465c-be3d-fce254debc65/volumes" Oct 30 20:17:36.049795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:36.049780 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" path="/var/lib/kubelet/pods/9a7e5628-1d20-497b-a673-d28a54b125f2/volumes" Oct 30 20:17:39.867527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.867493 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn"] Oct 30 20:17:39.867917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.867903 5733 cpu_manager.go:414] "containerMap: removing container" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" Oct 30 20:17:39.867980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.867917 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" Oct 30 20:17:39.868017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.867988 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7e5628-1d20-497b-a673-d28a54b125f2" containerName="discovery" Oct 30 20:17:39.871245 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.871227 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:39.876196 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.876172 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:17:39.876320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.876207 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:17:39.876320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.876210 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:39.876320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.876238 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:17:39.883922 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:39.883898 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn"] Oct 30 20:17:40.014786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.014756 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.014939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.014804 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.014939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.014914 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmgn\" (UniqueName: \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.115905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.115869 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.116117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.115935 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.116117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.116025 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmgn\" (UniqueName: \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.116536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.116514 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.118329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.118284 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.128502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.128473 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmgn\" (UniqueName: \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\") pod \"maas-api-revoke-sub-c1e271a9-682mn\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.186179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.186154 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:40.466501 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:40.466473 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn"] Oct 30 20:17:40.467768 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:17:40.467742 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb8b364_5bff_49f0_905e_d4d760ffd63c.slice/crio-1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490 WatchSource:0}: Error finding container 1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490: Status 404 returned error can't find the container with id 1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490 Oct 30 20:17:41.334385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:41.334353 5733 generic.go:356] "Generic (PLEG): container finished" podID="ddb8b364-5bff-49f0-905e-d4d760ffd63c" containerID="acd431761d02cfa642394d8a3077ac9b40350b8812a1226280e6850273e7e5a3" exitCode=0 Oct 30 20:17:41.334759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:41.334406 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" event={"ID":"ddb8b364-5bff-49f0-905e-d4d760ffd63c","Type":"ContainerDied","Data":"acd431761d02cfa642394d8a3077ac9b40350b8812a1226280e6850273e7e5a3"} Oct 30 20:17:41.334759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:41.334472 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" event={"ID":"ddb8b364-5bff-49f0-905e-d4d760ffd63c","Type":"ContainerStarted","Data":"1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490"} Oct 30 20:17:42.472811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.472786 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:42.646634 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.646600 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\") pod \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " Oct 30 20:17:42.646802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.646677 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\") pod \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " Oct 30 20:17:42.646802 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.646788 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\" (UniqueName: \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\") pod \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\" (UID: \"ddb8b364-5bff-49f0-905e-d4d760ffd63c\") " Oct 30 20:17:42.647084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.647028 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca" pod "ddb8b364-5bff-49f0-905e-d4d760ffd63c" (UID: "ddb8b364-5bff-49f0-905e-d4d760ffd63c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:42.648787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.648767 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca" pod "ddb8b364-5bff-49f0-905e-d4d760ffd63c" (UID: "ddb8b364-5bff-49f0-905e-d4d760ffd63c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:42.648903 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.648882 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn" pod "ddb8b364-5bff-49f0-905e-d4d760ffd63c" (UID: "ddb8b364-5bff-49f0-905e-d4d760ffd63c"). InnerVolumeSpecName "kube-api-access-cfmgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:42.747478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.747442 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfmgn\" (UniqueName: \"kubernetes.io/projected/ddb8b364-5bff-49f0-905e-d4d760ffd63c-kube-api-access-cfmgn\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:42.747478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.747472 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:42.747478 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:42.747482 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb8b364-5bff-49f0-905e-d4d760ffd63c-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:43.347268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:43.347225 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" event={"ID":"ddb8b364-5bff-49f0-905e-d4d760ffd63c","Type":"ContainerDied","Data":"1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490"} Oct 30 20:17:43.347268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:43.347260 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn" Oct 30 20:17:43.347477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:43.347291 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c0a9c5c4ab144a440025a9903cd4f0c95bac9dc50a736b41aa0cd8280706490" Oct 30 20:17:52.978286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:52.978238 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.316296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:17:54.316250 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:59.378725 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:17:59.378679 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:18:01.040864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.040829 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9"] Oct 30 20:18:01.041414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.041276 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ddb8b364-5bff-49f0-905e-d4d760ffd63c" containerName="revoke-keys" Oct 30 20:18:01.041414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.041290 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb8b364-5bff-49f0-905e-d4d760ffd63c" containerName="revoke-keys" Oct 30 20:18:01.041414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.041382 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb8b364-5bff-49f0-905e-d4d760ffd63c" containerName="revoke-keys" Oct 30 20:18:01.044508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.044487 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.054169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.054142 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:01.054169 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.054144 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:18:01.054360 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.054174 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:18:01.054420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.054405 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9"] Oct 30 20:18:01.054721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.054691 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:18:01.155734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.155701 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.155917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.155763 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.155917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.155882 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txf96\" (UniqueName: \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.256785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.256733 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.256979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.256847 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.256979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.256903 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txf96\" (UniqueName: \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.257595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.257562 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.259250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.259223 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.269763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.269734 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txf96\" (UniqueName: \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\") pod \"maas-api-revoke-sub-2e1f5b5f-4kgs9\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.360325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.360287 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:01.646115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:01.646081 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9"] Oct 30 20:18:01.647460 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:18:01.647432 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b786917_9b09_4e97_b649_7c0d77d966ea.slice/crio-114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f WatchSource:0}: Error finding container 114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f: Status 404 returned error can't find the container with id 114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f Oct 30 20:18:02.272556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:02.272472 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.422072 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:02.422026 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.487401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:02.487367 5733 generic.go:356] "Generic (PLEG): container finished" podID="8b786917-9b09-4e97-b649-7c0d77d966ea" containerID="0a979b2a45269ee59add7a3d5195466e7caa10228075542e2f6ec0b6db6c409e" exitCode=0 Oct 30 20:18:02.487591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:02.487439 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" event={"ID":"8b786917-9b09-4e97-b649-7c0d77d966ea","Type":"ContainerDied","Data":"0a979b2a45269ee59add7a3d5195466e7caa10228075542e2f6ec0b6db6c409e"} Oct 30 20:18:02.487591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:02.487480 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" event={"ID":"8b786917-9b09-4e97-b649-7c0d77d966ea","Type":"ContainerStarted","Data":"114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f"} Oct 30 20:18:03.627647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.627624 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:03.681652 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.681620 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\") pod \"8b786917-9b09-4e97-b649-7c0d77d966ea\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " Oct 30 20:18:03.681811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.681705 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\") pod \"8b786917-9b09-4e97-b649-7c0d77d966ea\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " Oct 30 20:18:03.681811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.681782 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\" (UniqueName: \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\") pod \"8b786917-9b09-4e97-b649-7c0d77d966ea\" (UID: \"8b786917-9b09-4e97-b649-7c0d77d966ea\") " Oct 30 20:18:03.682098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.682048 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca" pod "8b786917-9b09-4e97-b649-7c0d77d966ea" (UID: "8b786917-9b09-4e97-b649-7c0d77d966ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:03.682215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.682200 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:03.683766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.683735 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca" pod "8b786917-9b09-4e97-b649-7c0d77d966ea" (UID: "8b786917-9b09-4e97-b649-7c0d77d966ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:03.683870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.683814 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96" pod "8b786917-9b09-4e97-b649-7c0d77d966ea" (UID: "8b786917-9b09-4e97-b649-7c0d77d966ea"). InnerVolumeSpecName "kube-api-access-txf96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:03.783249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.783177 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b786917-9b09-4e97-b649-7c0d77d966ea-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:03.783249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:03.783205 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txf96\" (UniqueName: \"kubernetes.io/projected/8b786917-9b09-4e97-b649-7c0d77d966ea-kube-api-access-txf96\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:04.501197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:04.501166 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" Oct 30 20:18:04.501370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:04.501198 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9" event={"ID":"8b786917-9b09-4e97-b649-7c0d77d966ea","Type":"ContainerDied","Data":"114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f"} Oct 30 20:18:04.501370 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:04.501242 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="114a9143727ab7ca6ddff582915b3d49a79848aa084a56bff32fc2ee89bca57f" Oct 30 20:18:12.120860 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.120819 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6"] Oct 30 20:18:12.121296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.121265 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8b786917-9b09-4e97-b649-7c0d77d966ea" containerName="revoke-keys" Oct 30 20:18:12.121296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.121278 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b786917-9b09-4e97-b649-7c0d77d966ea" containerName="revoke-keys" Oct 30 20:18:12.121417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.121406 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b786917-9b09-4e97-b649-7c0d77d966ea" containerName="revoke-keys" Oct 30 20:18:12.124574 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.124555 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.129935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.129909 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:12.132143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.131685 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:18:12.132143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.131869 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:18:12.132143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.132073 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:18:12.137218 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.137194 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6"] Oct 30 20:18:12.261298 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.261255 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.261497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.261304 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk8rb\" (UniqueName: \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.261497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.261421 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.362936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.362902 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.363141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.362945 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk8rb\" (UniqueName: \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.363141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.362976 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.363607 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.363579 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.365419 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.365396 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.375629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.375574 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk8rb\" (UniqueName: \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\") pod \"maas-api-revoke-sub-ec843018-7t8b6\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.441916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.441883 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:12.727914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:12.727877 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6"] Oct 30 20:18:12.729379 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:18:12.729346 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b78e56_e8ff_47aa_8d55_ac3ca0eb3acd.slice/crio-9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e WatchSource:0}: Error finding container 9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e: Status 404 returned error can't find the container with id 9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e Oct 30 20:18:13.557560 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:13.557522 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" event={"ID":"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd","Type":"ContainerStarted","Data":"aef04fd1b659e8707acd45b16bea4ea5a4ca787fd8c5695b4c56eda8775ffc17"} Oct 30 20:18:13.557992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:13.557578 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" event={"ID":"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd","Type":"ContainerStarted","Data":"9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e"} Oct 30 20:18:13.578711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:13.578657 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" podStartSLOduration=1.578644607 podStartE2EDuration="1.578644607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:13.577296149 +0000 UTC m=+2853.645131708" watchObservedRunningTime="2026-10-30 20:18:13.578644607 +0000 UTC m=+2853.646480219" Oct 30 20:18:14.568425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:14.568390 5733 generic.go:356] "Generic (PLEG): container finished" podID="c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" containerID="aef04fd1b659e8707acd45b16bea4ea5a4ca787fd8c5695b4c56eda8775ffc17" exitCode=0 Oct 30 20:18:14.568980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:14.568461 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" event={"ID":"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd","Type":"ContainerDied","Data":"aef04fd1b659e8707acd45b16bea4ea5a4ca787fd8c5695b4c56eda8775ffc17"} Oct 30 20:18:16.217516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.217491 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:16.309795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.309761 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\") pod \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " Oct 30 20:18:16.309970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.309820 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\" (UniqueName: \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\") pod \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " Oct 30 20:18:16.309970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.309897 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\") pod \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\" (UID: \"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd\") " Oct 30 20:18:16.310182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.310155 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca" pod "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" (UID: "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:16.326270 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.326237 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb" pod "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" (UID: "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd"). InnerVolumeSpecName "kube-api-access-sk8rb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:16.331253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.331224 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca" pod "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" (UID: "c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:16.414920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.414883 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:16.415218 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.415195 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:16.415330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.415316 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk8rb\" (UniqueName: \"kubernetes.io/projected/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd-kube-api-access-sk8rb\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:16.629086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.629025 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" event={"ID":"c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd","Type":"ContainerDied","Data":"9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e"} Oct 30 20:18:16.629454 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.629432 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e238d5247c3ab0c7bf44ee9d5c7419784bc7ac4241c9bfa969b9a6948a81f7e" Oct 30 20:18:16.629640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:16.629358 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6" Oct 30 20:18:27.690266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:27.690221 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-z9kxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:27.690937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:27.690882 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:27.785083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:27.775183 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpgwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:18:27.785083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:27.775287 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:29.379463 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:18:29.379428 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:18:53.030173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:53.030081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:54.369627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:54.369589 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:55.441396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:55.441357 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:56.030858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:56.030822 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d"] Oct 30 20:18:56.035958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:56.035927 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d79c748-fh88d"] Oct 30 20:18:56.049179 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:18:56.049144 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee3e548-8002-448c-b440-b81408fef84d" path="/var/lib/kubelet/pods/5ee3e548-8002-448c-b440-b81408fef84d/volumes" Oct 30 20:18:59.378251 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:18:59.378214 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:19:22.991566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:22.991525 5733 scope.go:122] "RemoveContainer" containerID="1b57eee408478ffa2b56518386e31254d5f84a1fe38fc3d9a06c95e8930e0fcc" Oct 30 20:19:23.002746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:23.002720 5733 scope.go:122] "RemoveContainer" containerID="fb1302058f3c36f25e860812ab6cfe25a7057ef8b36a34a03b011a184eae6dc9" Oct 30 20:19:29.376579 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:19:29.376539 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:19:41.030112 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:41.030079 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n"] Oct 30 20:19:41.033429 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:41.033406 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d41b534d-z764n"] Oct 30 20:19:42.052746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:42.052698 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e06b756-f108-4a93-9ffa-825963305da1" path="/var/lib/kubelet/pods/4e06b756-f108-4a93-9ffa-825963305da1/volumes" Oct 30 20:19:51.026082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:51.026013 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv"] Oct 30 20:19:51.030127 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:51.030102 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd586ec8-ck5gv"] Oct 30 20:19:52.048333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:52.048298 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f94acb-3082-4e59-b92f-32b4fa49eef4" path="/var/lib/kubelet/pods/65f94acb-3082-4e59-b92f-32b4fa49eef4/volumes" Oct 30 20:19:53.088771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:53.088736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:54.421168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:19:54.421130 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:59.382589 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:19:59.382549 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:20:03.145230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:03.141556 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-v484k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:20:03.145230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:03.141639 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:20:23.040084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:23.039979 5733 scope.go:122] "RemoveContainer" containerID="c2567aea3b63134b5a9ab548fb4dced0a8c4cd240cb0eb95b81a266a368fcaaf" Oct 30 20:20:23.049284 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:23.049259 5733 scope.go:122] "RemoveContainer" containerID="7e56c72d18623217ef13e4f878c1c19ae981ab9d074d81c81db68c70fd33aede" Oct 30 20:20:29.504081 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:20:29.503389 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:20:40.919568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:40.919527 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:20:40.928457 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:40.928436 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:20:53.154228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:53.154188 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:54.339165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:54.339129 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:54.489537 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:54.489505 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:58.036844 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:58.036810 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb"] Oct 30 20:20:58.041012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:58.040983 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea564a8-5jtnb"] Oct 30 20:20:58.048673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:20:58.048639 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6891ecf-957f-4c8d-a833-ffc26ddc6789" path="/var/lib/kubelet/pods/e6891ecf-957f-4c8d-a833-ffc26ddc6789/volumes" Oct 30 20:20:59.380906 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:20:59.380853 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:21:23.078084 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:21:23.078029 5733 scope.go:122] "RemoveContainer" containerID="9e7e20d21800a27da71dade109d5d960f2918e86934e3046beeffc58e17c077c" Oct 30 20:21:23.086924 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:21:23.086902 5733 scope.go:122] "RemoveContainer" containerID="a0a586661deec210dbddeddd48593bfb8ff2da67ab118589b16fca91748fecb9" Oct 30 20:21:23.095602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:21:23.095582 5733 scope.go:122] "RemoveContainer" containerID="39dd1feac31e2b83ab034a54575e7df7f3f820de4b08434a286794b6f8cd030e" Oct 30 20:21:29.378153 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:21:29.378107 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:21:53.231816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:21:53.231736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.542939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:21:54.542901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:59.377342 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:21:59.377299 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:22:23.124709 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:23.124673 5733 scope.go:122] "RemoveContainer" containerID="e66c3f3e7400c2bf7b2343d3e99171cb242bc41f2eea0e36260d8612a658153c" Oct 30 20:22:23.133549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:23.133530 5733 scope.go:122] "RemoveContainer" containerID="529ee5ff6c40cd918bf1c91d7bcc57f8b0d916693e2307e0bed49b33dbd85b1e" Oct 30 20:22:23.141219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:23.141203 5733 scope.go:122] "RemoveContainer" containerID="c9cff19bc50619926bbb49d971aea6c4573bb36ed76b6990cc522fa17a90c208" Oct 30 20:22:29.376338 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:22:29.376292 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:22:43.027602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:43.027562 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn"] Oct 30 20:22:43.030287 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:43.030262 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-682mn"] Oct 30 20:22:44.049276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:44.049245 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb8b364-5bff-49f0-905e-d4d760ffd63c" path="/var/lib/kubelet/pods/ddb8b364-5bff-49f0-905e-d4d760ffd63c/volumes" Oct 30 20:22:53.252280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:53.252243 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:54.608766 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:22:54.608717 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:59.376279 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:22:59.376237 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:23:04.029036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:04.028995 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9"] Oct 30 20:23:04.034721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:04.034685 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4kgs9"] Oct 30 20:23:04.048489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:04.048460 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b786917-9b09-4e97-b649-7c0d77d966ea" path="/var/lib/kubelet/pods/8b786917-9b09-4e97-b649-7c0d77d966ea/volumes" Oct 30 20:23:07.355811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:07.355776 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:07.502555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:07.502523 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:16.057813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:16.057765 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6"] Oct 30 20:23:16.062982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:16.062960 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7t8b6"] Oct 30 20:23:18.048699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:18.048664 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" path="/var/lib/kubelet/pods/c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd/volumes" Oct 30 20:23:29.376703 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:23:29.376614 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:23:48.389910 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:48.389873 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:48.742535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:48.742458 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:53.305932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:53.305901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:54.650008 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:23:54.649969 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:59.376210 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:23:59.376165 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:24:23.187265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:24:23.187228 5733 scope.go:122] "RemoveContainer" containerID="acd431761d02cfa642394d8a3077ac9b40350b8812a1226280e6850273e7e5a3" Oct 30 20:24:23.196502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:24:23.196480 5733 scope.go:122] "RemoveContainer" containerID="aef04fd1b659e8707acd45b16bea4ea5a4ca787fd8c5695b4c56eda8775ffc17" Oct 30 20:24:23.204851 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:24:23.204828 5733 scope.go:122] "RemoveContainer" containerID="0a979b2a45269ee59add7a3d5195466e7caa10228075542e2f6ec0b6db6c409e" Oct 30 20:24:29.376668 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:24:29.376624 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:24:53.392221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:24:53.392138 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:54.812628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:24:54.812588 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:59.377143 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:24:59.377103 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:25:29.376434 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:25:29.376392 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:25:40.963637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:25:40.963606 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:25:40.977022 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:25:40.976998 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:25:53.476755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:25:53.476715 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:54.783709 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:25:54.783670 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:59.376710 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:25:59.376664 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:26:29.376843 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:29.376761 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:26:48.494403 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.494359 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:26:48.500334 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.495154 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" containerID="cri-o://9e9762e2cffd76d54afd0d88e8d52cf74b185a0081707eea14f63288975d3c65" gracePeriod=30 Oct 30 20:26:48.502229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.502207 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 20:26:48.502526 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.502504 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" containerID="cri-o://a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e" gracePeriod=30 Oct 30 20:26:48.504787 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.504761 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:26:48.504996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:48.504969 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" containerID="cri-o://8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537" gracePeriod=30 Oct 30 20:26:53.449604 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:53.449567 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:54.123357 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.123322 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:54.585221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.585194 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 20:26:54.634557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.634378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636040 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636118 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636168 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636213 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636244 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636359 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636398 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636426 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.636825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.636472 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\") pod \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\" (UID: \"99d64cc3-7d77-4972-8df4-4f7040a5de6f\") " Oct 30 20:26:54.639886 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.639837 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.640045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.639890 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.640045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.639892 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.641629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.641552 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:54.641629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.641559 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.641629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.641612 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.642250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.642209 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "kube-api-access-sh4rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:54.642885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.642860 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:54.649264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.649220 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token" pod "99d64cc3-7d77-4972-8df4-4f7040a5de6f" (UID: "99d64cc3-7d77-4972-8df4-4f7040a5de6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:54.737549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737515 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737541 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737551 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh4rv\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-kube-api-access-sh4rv\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737563 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737572 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737580 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737589 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737597 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99d64cc3-7d77-4972-8df4-4f7040a5de6f-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.737796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.737605 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99d64cc3-7d77-4972-8df4-4f7040a5de6f-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.766421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.766385 5733 generic.go:356] "Generic (PLEG): container finished" podID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerID="9e9762e2cffd76d54afd0d88e8d52cf74b185a0081707eea14f63288975d3c65" exitCode=0 Oct 30 20:26:54.766599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.766459 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" event={"ID":"a6eea678-8a13-45be-8ddd-010bf437f7dd","Type":"ContainerDied","Data":"9e9762e2cffd76d54afd0d88e8d52cf74b185a0081707eea14f63288975d3c65"} Oct 30 20:26:54.768260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.768228 5733 generic.go:356] "Generic (PLEG): container finished" podID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerID="a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e" exitCode=0 Oct 30 20:26:54.768414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.768285 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" Oct 30 20:26:54.768414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.768310 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" event={"ID":"99d64cc3-7d77-4972-8df4-4f7040a5de6f","Type":"ContainerDied","Data":"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e"} Oct 30 20:26:54.768414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.768374 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k" event={"ID":"99d64cc3-7d77-4972-8df4-4f7040a5de6f","Type":"ContainerDied","Data":"e57cd44d46e81107243d368c9734040f98c567b7b58014054d0e4986b7f79c51"} Oct 30 20:26:54.768414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.768403 5733 scope.go:122] "RemoveContainer" containerID="a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e" Oct 30 20:26:54.769923 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.769774 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:26:54.770686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.770661 5733 generic.go:356] "Generic (PLEG): container finished" podID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerID="8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537" exitCode=0 Oct 30 20:26:54.770811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.770707 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" event={"ID":"532bb3e4-bc28-438c-b4ae-5c8590d62be1","Type":"ContainerDied","Data":"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537"} Oct 30 20:26:54.770811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.770777 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" event={"ID":"532bb3e4-bc28-438c-b4ae-5c8590d62be1","Type":"ContainerDied","Data":"8c164e7c96b80d6438911d72cc66a148c547d3f6d891ecbde0c88aad8b6a1d50"} Oct 30 20:26:54.785300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.785276 5733 scope.go:122] "RemoveContainer" containerID="a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e" Oct 30 20:26:54.785653 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:54.785634 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e\": container with ID starting with a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e not found: ID does not exist" containerID="a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e" Oct 30 20:26:54.785756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.785666 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e"} err="failed to get container status \"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e\": rpc error: code = NotFound desc = could not find container \"a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e\": container with ID starting with a515dafff8777fc693bceeafeb9a19d775d64465193465c205fd977be921908e not found: ID does not exist" Oct 30 20:26:54.785756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.785690 5733 scope.go:122] "RemoveContainer" containerID="8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537" Oct 30 20:26:54.796418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.796399 5733 scope.go:122] "RemoveContainer" containerID="8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537" Oct 30 20:26:54.796738 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:54.796715 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537\": container with ID starting with 8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537 not found: ID does not exist" containerID="8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537" Oct 30 20:26:54.796808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.796749 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537"} err="failed to get container status \"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537\": rpc error: code = NotFound desc = could not find container \"8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537\": container with ID starting with 8b780e47b53aaa4c73cd70b5de95e4715a2fb9bd66e22c82e49b97451b56b537 not found: ID does not exist" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838653 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838715 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838779 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838832 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838875 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838916 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838961 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.838997 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.839030 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\") pod \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\" (UID: \"532bb3e4-bc28-438c-b4ae-5c8590d62be1\") " Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.840156 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.840187 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.840251 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.840432 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.840716 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.843018 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "kube-api-access-6ks57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.843513 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:54.845673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.845037 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:54.846813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.846139 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo" pod "532bb3e4-bc28-438c-b4ae-5c8590d62be1" (UID: "532bb3e4-bc28-438c-b4ae-5c8590d62be1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:54.846813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.846432 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 20:26:54.863912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.863884 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-v484k"] Oct 30 20:26:54.887831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.887750 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:54.940807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940762 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.940807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940802 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940816 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940829 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940843 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ks57\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-kube-api-access-6ks57\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940858 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940870 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940882 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:54.941077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:54.940894 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532bb3e4-bc28-438c-b4ae-5c8590d62be1-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.090226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.090189 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:55.153484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.153459 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:26:55.244362 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244325 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244380 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244415 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244503 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244546 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244582 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244642 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244679 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.244781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244713 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\") pod \"a6eea678-8a13-45be-8ddd-010bf437f7dd\" (UID: \"a6eea678-8a13-45be-8ddd-010bf437f7dd\") " Oct 30 20:26:55.245021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244778 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:55.245021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244843 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:55.245021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244798 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:55.245021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.244929 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:55.245223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.245080 5733 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-credential-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.245223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.245104 5733 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-data\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.245223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.245119 5733 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.245223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.245136 5733 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-workload-socket\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.245572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.245547 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:26:55.246816 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.246792 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:55.246958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.246937 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:26:55.247181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.247161 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "kube-api-access-gq4gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:26:55.247243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.247212 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo" pod "a6eea678-8a13-45be-8ddd-010bf437f7dd" (UID: "a6eea678-8a13-45be-8ddd-010bf437f7dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:26:55.345922 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.345886 5733 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-envoy\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.346126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.345937 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq4gj\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-kube-api-access-gq4gj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.346126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.345949 5733 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-podinfo\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.346126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.345958 5733 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eea678-8a13-45be-8ddd-010bf437f7dd-istio-token\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.346126 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.345967 5733 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eea678-8a13-45be-8ddd-010bf437f7dd-istiod-ca-cert\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:26:55.786726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.786687 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn" Oct 30 20:26:55.788544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.788512 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" event={"ID":"a6eea678-8a13-45be-8ddd-010bf437f7dd","Type":"ContainerDied","Data":"bb5984c29da714dafa54595216b3d0656d95086d73c9ee9397292c47ba0b4dee"} Oct 30 20:26:55.788696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.788524 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf" Oct 30 20:26:55.788696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.788606 5733 scope.go:122] "RemoveContainer" containerID="9e9762e2cffd76d54afd0d88e8d52cf74b185a0081707eea14f63288975d3c65" Oct 30 20:26:55.813041 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.813016 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:26:55.819841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.819818 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpgwn"] Oct 30 20:26:55.830926 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.830902 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:26:55.840549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:55.840516 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-z9kxf"] Oct 30 20:26:56.048468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:56.048380 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" path="/var/lib/kubelet/pods/532bb3e4-bc28-438c-b4ae-5c8590d62be1/volumes" Oct 30 20:26:56.048938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:56.048914 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" path="/var/lib/kubelet/pods/99d64cc3-7d77-4972-8df4-4f7040a5de6f/volumes" Oct 30 20:26:56.049426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:56.049407 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" path="/var/lib/kubelet/pods/a6eea678-8a13-45be-8ddd-010bf437f7dd/volumes" Oct 30 20:26:58.526658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:58.526631 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:58.696945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:58.696914 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:59.376288 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:59.376250 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:26:59.715129 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715002 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715469 5733 cpu_manager.go:414] "containerMap: removing container" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715484 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715509 5733 cpu_manager.go:414] "containerMap: removing container" podUID="c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" containerName="revoke-keys" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715514 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" containerName="revoke-keys" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715525 5733 cpu_manager.go:414] "containerMap: removing container" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715529 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" Oct 30 20:26:59.715540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715543 5733 cpu_manager.go:414] "containerMap: removing container" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" Oct 30 20:26:59.715781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715548 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" Oct 30 20:26:59.715781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715600 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b78e56-e8ff-47aa-8d55-ac3ca0eb3acd" containerName="revoke-keys" Oct 30 20:26:59.715781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715608 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d64cc3-7d77-4972-8df4-4f7040a5de6f" containerName="istio-proxy" Oct 30 20:26:59.715781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715617 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eea678-8a13-45be-8ddd-010bf437f7dd" containerName="istio-proxy" Oct 30 20:26:59.715781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.715625 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="532bb3e4-bc28-438c-b4ae-5c8590d62be1" containerName="istio-proxy" Oct 30 20:26:59.720496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.720469 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.740905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.740871 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:26:59.742110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.742084 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:26:59.742242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.742126 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:26:59.766313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.766284 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-69ac38-serving-cert\"" Oct 30 20:26:59.766462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.766436 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\"" Oct 30 20:26:59.766462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.766436 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:26:59.790398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.790369 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.790555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.790405 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.790555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.790501 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.790682 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.790604 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dztcz\" (UniqueName: \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.857555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.857520 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:26:59.891922 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.891889 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dztcz\" (UniqueName: \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.892107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.891944 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.892107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.891971 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.892107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.892025 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.892248 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:59.892154 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-serving-cert: secret "maas-api-e2e-worker-single-main-69ac38-serving-cert" not found Oct 30 20:26:59.892248 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:26:59.892219 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls podName:56271a58-226d-44ba-a824-5bc58b76bcc1 nodeName:}" failed. No retries permitted until 2026-10-30 20:27:00.392202588 +0000 UTC m=+3380.460038154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls") pod "maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" (UID: "56271a58-226d-44ba-a824-5bc58b76bcc1") : secret "maas-api-e2e-worker-single-main-69ac38-serving-cert" not found Oct 30 20:26:59.892715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.892690 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.894434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.894413 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.918960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.918929 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dztcz\" (UniqueName: \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:26:59.958351 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.958323 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:26:59.961997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.961980 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:26:59.991740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.991703 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-69ac38\"" Oct 30 20:26:59.993189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.993164 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:26:59.993407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.993380 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7c6f\" (UniqueName: \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:26:59.994380 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:26:59.994357 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:27:00.089313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.089279 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:27:00.092760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.092742 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.094166 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.094141 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7c6f\" (UniqueName: \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.094274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.094257 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.094949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.094925 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.099979 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.099955 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-c24eb963-dockercfg-bm7gd\"" Oct 30 20:27:00.113192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.113171 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:27:00.116841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.116823 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7c6f\" (UniqueName: \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\") pod \"payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.126577 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.126556 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.195136 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.195109 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jtqc\" (UniqueName: \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.195296 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.195181 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.274994 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.274919 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.296040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.296015 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jtqc\" (UniqueName: \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.296171 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.296131 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.296797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.296774 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.314372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.314351 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jtqc\" (UniqueName: \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\") pod \"payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.397303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.397264 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:27:00.399603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.399575 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:27:00.406463 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.406440 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.560188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.560161 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:27:00.562331 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:27:00.562295 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837e39c3_dd23_4014_808a_5b8d595f7bc8.slice/crio-1850e1f2cafb2b07037b54aa512e4ec48df1b2b710455e6fa0d10f6f15cbc2bf WatchSource:0}: Error finding container 1850e1f2cafb2b07037b54aa512e4ec48df1b2b710455e6fa0d10f6f15cbc2bf: Status 404 returned error can't find the container with id 1850e1f2cafb2b07037b54aa512e4ec48df1b2b710455e6fa0d10f6f15cbc2bf Oct 30 20:27:00.635901 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.635875 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:27:00.688988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.688957 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:27:00.696490 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:27:00.695464 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2123a4cd_f0a5_4ea8_be0b_4e56f599fd7f.slice/crio-b0c9d3af56b470417e023a8bbc15808cd52a4d2cdc4d3a1fde015fbd8222ed0c WatchSource:0}: Error finding container b0c9d3af56b470417e023a8bbc15808cd52a4d2cdc4d3a1fde015fbd8222ed0c: Status 404 returned error can't find the container with id b0c9d3af56b470417e023a8bbc15808cd52a4d2cdc4d3a1fde015fbd8222ed0c Oct 30 20:27:00.820464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.820369 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" event={"ID":"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f","Type":"ContainerStarted","Data":"4253e96b145023c2cc0bef0ff09f12c1ecde01c83f9cbd3f2707cdf2e27bcd76"} Oct 30 20:27:00.820464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.820439 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" event={"ID":"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f","Type":"ContainerStarted","Data":"b0c9d3af56b470417e023a8bbc15808cd52a4d2cdc4d3a1fde015fbd8222ed0c"} Oct 30 20:27:00.820954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.820544 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:00.822067 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.822026 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" event={"ID":"837e39c3-dd23-4014-808a-5b8d595f7bc8","Type":"ContainerStarted","Data":"57292b70c4ad1d026c087657e100edfb1d5a254e5ffbdf25fb5aa16053e4608f"} Oct 30 20:27:00.822187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.822107 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:00.822187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.822137 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" event={"ID":"837e39c3-dd23-4014-808a-5b8d595f7bc8","Type":"ContainerStarted","Data":"1850e1f2cafb2b07037b54aa512e4ec48df1b2b710455e6fa0d10f6f15cbc2bf"} Oct 30 20:27:00.840605 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.840561 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" podStartSLOduration=0.840550558 podStartE2EDuration="840.550558ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:00.837016697 +0000 UTC m=+3380.904852265" watchObservedRunningTime="2026-10-30 20:27:00.840550558 +0000 UTC m=+3380.908386134" Oct 30 20:27:00.857194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.857149 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" podStartSLOduration=1.857136094 podStartE2EDuration="1.857136094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:00.855646009 +0000 UTC m=+3380.923481578" watchObservedRunningTime="2026-10-30 20:27:00.857136094 +0000 UTC m=+3380.924971664" Oct 30 20:27:00.900359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.900330 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:00.921301 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.921273 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:27:00.921994 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:27:00.921970 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56271a58_226d_44ba_a824_5bc58b76bcc1.slice/crio-0cb96ef7edf6cd38a240b2053b00713970cbc5a361a2a1c6309aea61b55a81f5 WatchSource:0}: Error finding container 0cb96ef7edf6cd38a240b2053b00713970cbc5a361a2a1c6309aea61b55a81f5: Status 404 returned error can't find the container with id 0cb96ef7edf6cd38a240b2053b00713970cbc5a361a2a1c6309aea61b55a81f5 Oct 30 20:27:00.923958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:00.923938 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:27:01.830932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:01.830888 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" event={"ID":"56271a58-226d-44ba-a824-5bc58b76bcc1","Type":"ContainerStarted","Data":"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b"} Oct 30 20:27:01.831411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:01.830971 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:27:01.831411 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:01.830995 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" event={"ID":"56271a58-226d-44ba-a824-5bc58b76bcc1","Type":"ContainerStarted","Data":"0cb96ef7edf6cd38a240b2053b00713970cbc5a361a2a1c6309aea61b55a81f5"} Oct 30 20:27:01.839407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:01.839358 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:01.855325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:01.855261 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" podStartSLOduration=2.120505443 podStartE2EDuration="2.855241292s" totalImagesPullingTime="734.735849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:01.851378195 +0000 UTC m=+3381.919213764" watchObservedRunningTime="2026-10-30 20:27:01.855241292 +0000 UTC m=+3381.923076861" Oct 30 20:27:02.732233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:02.732192 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:03.591847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:03.591814 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:04.491974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:04.491939 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:05.292995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:05.292963 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:06.098784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:06.098751 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:06.891741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:06.891689 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:07.705892 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:07.705857 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:07.844168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:07.844136 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:27:08.516286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:08.516230 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:09.426992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:09.426964 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:10.302401 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:10.302345 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.083170 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:11.083135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:11.834388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:11.834359 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:27:11.834824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:11.834437 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:27:11.903585 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:11.903541 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:12.691835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:12.691804 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:13.716028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:13.715996 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:13.883620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:13.883583 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:14.242955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:14.242910 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:14.394965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:14.394929 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:14.553399 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:14.553295 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:15.575148 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:15.575116 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:15.718540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:15.718506 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:16.600532 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:16.600502 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:17.551576 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:17.551535 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.407795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.407761 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:18.539851 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.539819 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9"] Oct 30 20:27:18.543293 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.543276 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.548664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.548634 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-fbh7r\"" Oct 30 20:27:18.549073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.549035 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 20:27:18.566685 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.566628 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9"] Oct 30 20:27:18.673862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.673755 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.673884 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.673967 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674014 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674069 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674036 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674089 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674191 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674253 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.674499 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.674323 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbv26\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-kube-api-access-nbv26\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775486 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775447 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775499 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775538 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775556 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbv26\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-kube-api-access-nbv26\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775582 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775614 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.775647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775638 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775680 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776027 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.775714 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.776021 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.776377 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.776469 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.776480 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.776932 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.776911 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.778524 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.778501 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.778835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.778813 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.796586 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.796556 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbv26\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-kube-api-access-nbv26\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.797450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.797430 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393595d7-bbd7-4db8-b5a5-5db2cc93b72b-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-ldpp9\" (UID: \"393595d7-bbd7-4db8-b5a5-5db2cc93b72b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:18.858323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:18.858290 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:19.147623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.147580 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9"] Oct 30 20:27:19.150663 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:27:19.150631 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393595d7_bbd7_4db8_b5a5_5db2cc93b72b.slice/crio-a57ab8dd12d5b1ac14c9fdeb658e18e85980d10f8db3f2d43b0d803d8a6b1c30 WatchSource:0}: Error finding container a57ab8dd12d5b1ac14c9fdeb658e18e85980d10f8db3f2d43b0d803d8a6b1c30: Status 404 returned error can't find the container with id a57ab8dd12d5b1ac14c9fdeb658e18e85980d10f8db3f2d43b0d803d8a6b1c30 Oct 30 20:27:19.153230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.153194 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:27:19.153318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.153280 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:27:19.153318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.153313 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:27:19.461128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.461011 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:19.949813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.949770 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" event={"ID":"393595d7-bbd7-4db8-b5a5-5db2cc93b72b","Type":"ContainerStarted","Data":"92074487819e8c49b1fc2889c64736fb27861637b87bc15f9be43bb5292c2957"} Oct 30 20:27:19.950030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.949829 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" event={"ID":"393595d7-bbd7-4db8-b5a5-5db2cc93b72b","Type":"ContainerStarted","Data":"a57ab8dd12d5b1ac14c9fdeb658e18e85980d10f8db3f2d43b0d803d8a6b1c30"} Oct 30 20:27:19.975587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:19.975533 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podStartSLOduration=1.9755147929999999 podStartE2EDuration="1.975514793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:27:19.972996145 +0000 UTC m=+3400.040831714" watchObservedRunningTime="2026-10-30 20:27:19.975514793 +0000 UTC m=+3400.043350392" Oct 30 20:27:20.436857 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:20.436799 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:20.859409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:20.859378 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:20.860463 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:20.860440 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:20.860536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:20.860492 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:21.423725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:21.423687 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:21.859425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:21.859395 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:21.859808 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:21.859441 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:22.336489 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:22.336397 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:22.858751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:22.858716 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:22.858916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:22.858776 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:23.336008 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:23.335972 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:23.858918 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:23.858881 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:23.859128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:23.858936 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:24.265756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:24.265669 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:24.859358 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:24.859322 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:24.859767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:24.859394 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:25.113958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:25.113867 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:25.858974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:25.858942 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-ldpp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" start-of-body= Oct 30 20:27:25.859174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:25.858991 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" podUID="393595d7-bbd7-4db8-b5a5-5db2cc93b72b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.218:15021/healthz/ready\": dial tcp 10.133.0.218:15021: connect: connection refused" Oct 30 20:27:26.004564 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:26.004533 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:26.796503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:26.796474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:27.451773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:27.451738 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:27.452276 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:27.452137 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:27.453141 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:27.453120 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-ldpp9" Oct 30 20:27:27.628189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:27.628151 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:28.474723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:28.474694 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:28.622796 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:28.622763 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:29.287630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:29.287594 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:29.376298 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:27:29.376262 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:27:30.092571 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:30.092537 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:31.010073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:31.010029 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:31.895143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:31.895112 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:32.694647 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:32.694616 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:33.498545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:33.498494 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:34.402943 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:34.402899 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:35.465958 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:35.465923 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:36.400949 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:36.400910 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:37.277621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:37.277584 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:38.091372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:38.091324 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:39.091712 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:39.091679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:39.879047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:39.879008 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:40.689254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:40.689215 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:41.495960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:41.495922 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:42.295232 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:42.295194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:43.101095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:43.101049 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:43.997811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:43.997781 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:44.919944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:44.919909 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:45.693152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:45.693050 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:46.669891 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:46.669857 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:47.375965 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:47.375935 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:48.185702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:48.185659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:48.984566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:48.984531 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:50.012311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:50.012275 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:50.989202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:50.989124 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:51.796303 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:51.796240 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:52.584440 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:52.584407 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.386291 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:53.386252 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:53.650566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:53.650491 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:54.286636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:54.286604 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:54.893708 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:54.893668 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:55.131521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:55.131488 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:55.930644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:55.930608 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:56.806913 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:56.806879 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:56.955264 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:56.955234 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:57.854732 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:57.854703 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:58.886272 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:58.886238 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:27:59.377920 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:27:59.377878 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:27:59.682186 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:27:59.682081 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:00.485627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:00.485573 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.288431 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:01.288403 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:01.968629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:01.968594 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:02.192954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:02.192920 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:03.285286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:03.285234 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:04.082386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:04.082355 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:04.902398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:04.902364 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:05.790813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:05.790778 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:06.605799 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:06.605757 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:07.089749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:07.089662 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:28:07.109674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:07.109637 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:28:07.209234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:07.209201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:07.475675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:07.475642 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:07.614490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:07.614457 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.195124 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.195087 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:28:08.195364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.195338 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" podUID="56271a58-226d-44ba-a824-5bc58b76bcc1" containerName="maas-api" containerID="cri-o://6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b" gracePeriod=30 Oct 30 20:28:08.283974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.283941 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.447427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.447371 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:28:08.469620 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.469593 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.583108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583070 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\") pod \"56271a58-226d-44ba-a824-5bc58b76bcc1\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " Oct 30 20:28:08.583324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583131 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\") pod \"56271a58-226d-44ba-a824-5bc58b76bcc1\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " Oct 30 20:28:08.583324 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583292 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") pod \"56271a58-226d-44ba-a824-5bc58b76bcc1\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " Oct 30 20:28:08.583519 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583326 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\" (UniqueName: \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\") pod \"56271a58-226d-44ba-a824-5bc58b76bcc1\" (UID: \"56271a58-226d-44ba-a824-5bc58b76bcc1\") " Oct 30 20:28:08.583589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583515 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca" pod "56271a58-226d-44ba-a824-5bc58b76bcc1" (UID: "56271a58-226d-44ba-a824-5bc58b76bcc1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:08.583687 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.583668 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56271a58-226d-44ba-a824-5bc58b76bcc1-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:08.585639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.585615 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz" pod "56271a58-226d-44ba-a824-5bc58b76bcc1" (UID: "56271a58-226d-44ba-a824-5bc58b76bcc1"). InnerVolumeSpecName "kube-api-access-dztcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:08.585763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.585712 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs" pod "56271a58-226d-44ba-a824-5bc58b76bcc1" (UID: "56271a58-226d-44ba-a824-5bc58b76bcc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:08.585835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.585809 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls" pod "56271a58-226d-44ba-a824-5bc58b76bcc1" (UID: "56271a58-226d-44ba-a824-5bc58b76bcc1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:08.611340 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.611309 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:08.685252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.685222 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:08.685252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.685250 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56271a58-226d-44ba-a824-5bc58b76bcc1-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:08.685641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:08.685263 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dztcz\" (UniqueName: \"kubernetes.io/projected/56271a58-226d-44ba-a824-5bc58b76bcc1-kube-api-access-dztcz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:09.072029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.071985 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:09.227970 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.227930 5733 generic.go:356] "Generic (PLEG): container finished" podID="56271a58-226d-44ba-a824-5bc58b76bcc1" containerID="6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b" exitCode=0 Oct 30 20:28:09.228200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.227986 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" Oct 30 20:28:09.228200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.228014 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" event={"ID":"56271a58-226d-44ba-a824-5bc58b76bcc1","Type":"ContainerDied","Data":"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b"} Oct 30 20:28:09.228200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.228103 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m" event={"ID":"56271a58-226d-44ba-a824-5bc58b76bcc1","Type":"ContainerDied","Data":"0cb96ef7edf6cd38a240b2053b00713970cbc5a361a2a1c6309aea61b55a81f5"} Oct 30 20:28:09.228200 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.228135 5733 scope.go:122] "RemoveContainer" containerID="6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b" Oct 30 20:28:09.237960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.237938 5733 scope.go:122] "RemoveContainer" containerID="6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b" Oct 30 20:28:09.238240 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:28:09.238220 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b\": container with ID starting with 6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b not found: ID does not exist" containerID="6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b" Oct 30 20:28:09.238308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.238250 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b"} err="failed to get container status \"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b\": rpc error: code = NotFound desc = could not find container \"6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b\": container with ID starting with 6159344c7005e22525afd175cfdb93c02f75b2a4f3493664758cf1401206178b not found: ID does not exist" Oct 30 20:28:09.251948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.251925 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:28:09.254441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:09.254421 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-69ac38-66b59f84bd-m9x8m"] Oct 30 20:28:10.049921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:10.049883 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56271a58-226d-44ba-a824-5bc58b76bcc1" path="/var/lib/kubelet/pods/56271a58-226d-44ba-a824-5bc58b76bcc1/volumes" Oct 30 20:28:12.090976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:12.090934 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" containerID="cri-o://4253e96b145023c2cc0bef0ff09f12c1ecde01c83f9cbd3f2707cdf2e27bcd76" gracePeriod=25 Oct 30 20:28:12.110311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:12.110273 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" containerID="cri-o://57292b70c4ad1d026c087657e100edfb1d5a254e5ffbdf25fb5aa16053e4608f" gracePeriod=25 Oct 30 20:28:15.302251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:15.302219 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.462551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:15.462514 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:15.664330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:15.664282 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:16.976721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.976682 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99"] Oct 30 20:28:16.977226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.977147 5733 cpu_manager.go:414] "containerMap: removing container" podUID="56271a58-226d-44ba-a824-5bc58b76bcc1" containerName="maas-api" Oct 30 20:28:16.977226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.977163 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56271a58-226d-44ba-a824-5bc58b76bcc1" containerName="maas-api" Oct 30 20:28:16.977308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.977251 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="56271a58-226d-44ba-a824-5bc58b76bcc1" containerName="maas-api" Oct 30 20:28:16.980643 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.980626 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:16.985891 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.985784 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:28:16.986120 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.985905 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:28:16.986459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.986433 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:28:16.986983 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.986960 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:28:16.990252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:16.990211 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99"] Oct 30 20:28:17.066847 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.066805 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.067038 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.066860 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.067117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.067081 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xn77\" (UniqueName: \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.168335 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.168294 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xn77\" (UniqueName: \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.168581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.168396 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.168581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.168445 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.169543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.169505 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.171331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.171302 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.185933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.185899 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xn77\" (UniqueName: \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\") pod \"maas-api-revoke-sub-3d4156f1-5rd99\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.296490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.296411 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:17.577974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:17.577949 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99"] Oct 30 20:28:17.579820 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:28:17.579776 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c738da4_b62d_4a98_9dab_2054fde8ecc2.slice/crio-17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40 WatchSource:0}: Error finding container 17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40: Status 404 returned error can't find the container with id 17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40 Oct 30 20:28:18.281533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:18.281495 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" event={"ID":"4c738da4-b62d-4a98-9dab-2054fde8ecc2","Type":"ContainerStarted","Data":"c4535ca01be9b7bb0f6dc75339ef29b8fa78ba3f29590a27f1ba2449f2c0e967"} Oct 30 20:28:18.281923 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:18.281552 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" event={"ID":"4c738da4-b62d-4a98-9dab-2054fde8ecc2","Type":"ContainerStarted","Data":"17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40"} Oct 30 20:28:18.301630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:18.301583 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" podStartSLOduration=2.301568826 podStartE2EDuration="2.301568826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:18.29822135 +0000 UTC m=+3458.366056918" watchObservedRunningTime="2026-10-30 20:28:18.301568826 +0000 UTC m=+3458.369404393" Oct 30 20:28:19.288786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:19.288752 5733 generic.go:356] "Generic (PLEG): container finished" podID="4c738da4-b62d-4a98-9dab-2054fde8ecc2" containerID="c4535ca01be9b7bb0f6dc75339ef29b8fa78ba3f29590a27f1ba2449f2c0e967" exitCode=0 Oct 30 20:28:19.289224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:19.288840 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" event={"ID":"4c738da4-b62d-4a98-9dab-2054fde8ecc2","Type":"ContainerDied","Data":"c4535ca01be9b7bb0f6dc75339ef29b8fa78ba3f29590a27f1ba2449f2c0e967"} Oct 30 20:28:20.426197 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.426174 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:20.495130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.495097 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\") pod \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " Oct 30 20:28:20.495304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.495139 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\" (UniqueName: \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\") pod \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " Oct 30 20:28:20.495304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.495218 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\") pod \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\" (UID: \"4c738da4-b62d-4a98-9dab-2054fde8ecc2\") " Oct 30 20:28:20.495608 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.495585 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca" pod "4c738da4-b62d-4a98-9dab-2054fde8ecc2" (UID: "4c738da4-b62d-4a98-9dab-2054fde8ecc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:20.497425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.497396 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca" pod "4c738da4-b62d-4a98-9dab-2054fde8ecc2" (UID: "4c738da4-b62d-4a98-9dab-2054fde8ecc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:28:20.497547 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.497402 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77" pod "4c738da4-b62d-4a98-9dab-2054fde8ecc2" (UID: "4c738da4-b62d-4a98-9dab-2054fde8ecc2"). InnerVolumeSpecName "kube-api-access-9xn77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:20.596468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.596435 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:20.596468 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.596469 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c738da4-b62d-4a98-9dab-2054fde8ecc2-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:20.596684 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:20.596484 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xn77\" (UniqueName: \"kubernetes.io/projected/4c738da4-b62d-4a98-9dab-2054fde8ecc2-kube-api-access-9xn77\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:21.300365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:21.300335 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" Oct 30 20:28:21.300545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:21.300367 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99" event={"ID":"4c738da4-b62d-4a98-9dab-2054fde8ecc2","Type":"ContainerDied","Data":"17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40"} Oct 30 20:28:21.300545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:21.300416 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b116e5ecfa63fea2618442cf65629a195c62b835bdd8e88b2a6975116ffb40" Oct 30 20:28:21.833202 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:21.833167 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:21.833674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:21.833161 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:23.037239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:23.037205 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:29.377298 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:28:29.377250 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:28:31.832611 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:31.832568 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:31.833052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:31.832572 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:28:32.370504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.370297 5733 generic.go:356] "Generic (PLEG): container finished" podID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerID="4253e96b145023c2cc0bef0ff09f12c1ecde01c83f9cbd3f2707cdf2e27bcd76" exitCode=0 Oct 30 20:28:32.370504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.370407 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" event={"ID":"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f","Type":"ContainerDied","Data":"4253e96b145023c2cc0bef0ff09f12c1ecde01c83f9cbd3f2707cdf2e27bcd76"} Oct 30 20:28:32.374552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.374512 5733 generic.go:356] "Generic (PLEG): container finished" podID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerID="57292b70c4ad1d026c087657e100edfb1d5a254e5ffbdf25fb5aa16053e4608f" exitCode=0 Oct 30 20:28:32.374686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.374625 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" event={"ID":"837e39c3-dd23-4014-808a-5b8d595f7bc8","Type":"ContainerDied","Data":"57292b70c4ad1d026c087657e100edfb1d5a254e5ffbdf25fb5aa16053e4608f"} Oct 30 20:28:32.401561 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.401534 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:28:32.440562 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.440255 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:28:32.515437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515396 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\") pod \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " Oct 30 20:28:32.515644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515450 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\" (UniqueName: \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\") pod \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\" (UID: \"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f\") " Oct 30 20:28:32.515644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515489 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\" (UniqueName: \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\") pod \"837e39c3-dd23-4014-808a-5b8d595f7bc8\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " Oct 30 20:28:32.515644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515583 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\") pod \"837e39c3-dd23-4014-808a-5b8d595f7bc8\" (UID: \"837e39c3-dd23-4014-808a-5b8d595f7bc8\") " Oct 30 20:28:32.515867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515837 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume" pod "2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" (UID: "2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:32.515977 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.515957 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:32.516254 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.516225 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume" pod "837e39c3-dd23-4014-808a-5b8d595f7bc8" (UID: "837e39c3-dd23-4014-808a-5b8d595f7bc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:28:32.518748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.518725 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f" pod "837e39c3-dd23-4014-808a-5b8d595f7bc8" (UID: "837e39c3-dd23-4014-808a-5b8d595f7bc8"). InnerVolumeSpecName "kube-api-access-b7c6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:32.519749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.519718 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc" pod "2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" (UID: "2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f"). InnerVolumeSpecName "kube-api-access-5jtqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:32.617125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.617082 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/837e39c3-dd23-4014-808a-5b8d595f7bc8-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:32.617125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.617114 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jtqc\" (UniqueName: \"kubernetes.io/projected/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f-kube-api-access-5jtqc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:32.617125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:32.617128 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7c6f\" (UniqueName: \"kubernetes.io/projected/837e39c3-dd23-4014-808a-5b8d595f7bc8-kube-api-access-b7c6f\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:33.382763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.382679 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" Oct 30 20:28:33.382763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.382678 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh" event={"ID":"2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f","Type":"ContainerDied","Data":"b0c9d3af56b470417e023a8bbc15808cd52a4d2cdc4d3a1fde015fbd8222ed0c"} Oct 30 20:28:33.383450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.382819 5733 scope.go:122] "RemoveContainer" containerID="4253e96b145023c2cc0bef0ff09f12c1ecde01c83f9cbd3f2707cdf2e27bcd76" Oct 30 20:28:33.384437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.384264 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" Oct 30 20:28:33.384437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.384263 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7" event={"ID":"837e39c3-dd23-4014-808a-5b8d595f7bc8","Type":"ContainerDied","Data":"1850e1f2cafb2b07037b54aa512e4ec48df1b2b710455e6fa0d10f6f15cbc2bf"} Oct 30 20:28:33.393912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.393896 5733 scope.go:122] "RemoveContainer" containerID="57292b70c4ad1d026c087657e100edfb1d5a254e5ffbdf25fb5aa16053e4608f" Oct 30 20:28:33.407367 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.407346 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:28:33.412274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.412253 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-69ac38-776455659nxlrh"] Oct 30 20:28:33.422928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.422905 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:28:33.429786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:33.429767 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-69ac38-574cb28qx7"] Oct 30 20:28:34.049248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:34.049212 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" path="/var/lib/kubelet/pods/2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f/volumes" Oct 30 20:28:34.049783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:34.049762 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" path="/var/lib/kubelet/pods/837e39c3-dd23-4014-808a-5b8d595f7bc8/volumes" Oct 30 20:28:35.812313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:35.812276 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 20:28:35.812771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:35.812500 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" containerID="cri-o://336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e" gracePeriod=10 Oct 30 20:28:36.080748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.080724 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:28:36.151919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.151881 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\") pod \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " Oct 30 20:28:36.152165 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.152076 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\" (UniqueName: \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\") pod \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\" (UID: \"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2\") " Oct 30 20:28:36.152754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.152327 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume" pod "6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" (UID: "6049f1ce-d07c-4f37-9a3c-dfeff0232cf2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:28:36.152754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.152508 5733 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-extensions-socket-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:36.154575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.154548 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r" pod "6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" (UID: "6049f1ce-d07c-4f37-9a3c-dfeff0232cf2"). InnerVolumeSpecName "kube-api-access-7rg4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:28:36.253717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.253675 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rg4r\" (UniqueName: \"kubernetes.io/projected/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2-kube-api-access-7rg4r\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:28:36.403582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.403550 5733 generic.go:356] "Generic (PLEG): container finished" podID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerID="336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e" exitCode=0 Oct 30 20:28:36.403772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.403609 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" Oct 30 20:28:36.403772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.403652 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerDied","Data":"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e"} Oct 30 20:28:36.403772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.403716 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc" event={"ID":"6049f1ce-d07c-4f37-9a3c-dfeff0232cf2","Type":"ContainerDied","Data":"23aff50cfa5c4219ce441dce1483254829a033ba8fab5f22cb61fa99c1a68210"} Oct 30 20:28:36.403772 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.403746 5733 scope.go:122] "RemoveContainer" containerID="336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e" Oct 30 20:28:36.413791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.413773 5733 scope.go:122] "RemoveContainer" containerID="666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6" Oct 30 20:28:36.422099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.422080 5733 scope.go:122] "RemoveContainer" containerID="336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e" Oct 30 20:28:36.422378 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:28:36.422359 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e\": container with ID starting with 336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e not found: ID does not exist" containerID="336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e" Oct 30 20:28:36.422428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.422388 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e"} err="failed to get container status \"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e\": rpc error: code = NotFound desc = could not find container \"336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e\": container with ID starting with 336fbade0519c50a267b10f3e4464f44be75f5572d2b07ac2592f8458026278e not found: ID does not exist" Oct 30 20:28:36.422428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.422403 5733 scope.go:122] "RemoveContainer" containerID="666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6" Oct 30 20:28:36.422636 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:28:36.422617 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6\": container with ID starting with 666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6 not found: ID does not exist" containerID="666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6" Oct 30 20:28:36.422673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.422641 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6"} err="failed to get container status \"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6\": rpc error: code = NotFound desc = could not find container \"666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6\": container with ID starting with 666edcbbbc5fe1b5654a73922f98ec3bd7233b9ad29ed51b53c08e19d3b835c6 not found: ID does not exist" Oct 30 20:28:36.427381 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.427359 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 20:28:36.432106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:36.432080 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-rcvnc"] Oct 30 20:28:38.048630 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:38.048598 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" path="/var/lib/kubelet/pods/6049f1ce-d07c-4f37-9a3c-dfeff0232cf2/volumes" Oct 30 20:28:39.342053 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:39.342018 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:39.710944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:39.710910 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:45.382864 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:45.382831 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:48.654464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.654428 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s"] Oct 30 20:28:48.655495 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655470 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.655639 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655626 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.655744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655732 5733 cpu_manager.go:414] "containerMap: removing container" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" Oct 30 20:28:48.655817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655807 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" Oct 30 20:28:48.655906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655897 5733 cpu_manager.go:414] "containerMap: removing container" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" Oct 30 20:28:48.655995 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.655967 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" Oct 30 20:28:48.656108 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656097 5733 cpu_manager.go:414] "containerMap: removing container" podUID="4c738da4-b62d-4a98-9dab-2054fde8ecc2" containerName="revoke-keys" Oct 30 20:28:48.656189 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656180 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c738da4-b62d-4a98-9dab-2054fde8ecc2" containerName="revoke-keys" Oct 30 20:28:48.656269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656261 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.656336 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656328 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.656516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656505 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="2123a4cd-f0a5-4ea8-be0b-4e56f599fd7f" containerName="payload-processing" Oct 30 20:28:48.656593 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656584 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.656670 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656661 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c738da4-b62d-4a98-9dab-2054fde8ecc2" containerName="revoke-keys" Oct 30 20:28:48.656744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656736 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.656813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656805 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.656945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.656932 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="837e39c3-dd23-4014-808a-5b8d595f7bc8" containerName="payload-pre-processing" Oct 30 20:28:48.680090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.679296 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s"] Oct 30 20:28:48.680090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.679465 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.682226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.682199 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn"] Oct 30 20:28:48.682762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.682738 5733 cpu_manager.go:414] "containerMap: removing container" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.682873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.682763 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6049f1ce-d07c-4f37-9a3c-dfeff0232cf2" containerName="manager" Oct 30 20:28:48.686277 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.686257 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.696288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.696261 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn"] Oct 30 20:28:48.778917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.778880 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.778917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.778919 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.778936 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779046 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779117 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779139 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32aa4d2b-e4fa-4a68-883e-082788e51c95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779184 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779209 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjsw\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-kube-api-access-xwjsw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779224 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779277 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tnf4\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-kube-api-access-2tnf4\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779298 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779316 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779342 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779464 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779394 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779504 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779525 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.779569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779549 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.779668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.779612 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880113 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880050 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880126 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880151 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880182 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880220 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880245 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32aa4d2b-e4fa-4a68-883e-082788e51c95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880275 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880310 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjsw\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-kube-api-access-xwjsw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880333 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880368 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tnf4\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-kube-api-access-2tnf4\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880396 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880426 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880468 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880507 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880541 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880573 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880586 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880609 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880660 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.880776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880693 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880779 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.880900 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881004 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881033 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881185 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881262 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.881552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881461 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.881980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.881633 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32aa4d2b-e4fa-4a68-883e-082788e51c95-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.882992 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.882964 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.883134 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.883111 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.883339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.883323 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.883504 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.883487 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.891642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.891618 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:48.891727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.891669 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjsw\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-kube-api-access-xwjsw\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.892111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.892093 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-8fcjn\" (UID: \"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:48.892657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:48.892640 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tnf4\" (UniqueName: \"kubernetes.io/projected/32aa4d2b-e4fa-4a68-883e-082788e51c95-kube-api-access-2tnf4\") pod \"maas-default-gateway-openshift-default-57f78bcd55-qr62s\" (UID: \"32aa4d2b-e4fa-4a68-883e-082788e51c95\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:49.002174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.002105 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:49.010215 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.010193 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:49.289431 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.289396 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s"] Oct 30 20:28:49.290988 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:28:49.290958 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aa4d2b_e4fa_4a68_883e_082788e51c95.slice/crio-92eb246a38fec7b6a305d4847c56aadd995e8a59e139b31f6272a9b70ef4ecb3 WatchSource:0}: Error finding container 92eb246a38fec7b6a305d4847c56aadd995e8a59e139b31f6272a9b70ef4ecb3: Status 404 returned error can't find the container with id 92eb246a38fec7b6a305d4847c56aadd995e8a59e139b31f6272a9b70ef4ecb3 Oct 30 20:28:49.293421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.293372 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.293516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.293457 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.293516 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.293488 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.300378 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.297932 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn"] Oct 30 20:28:49.303006 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:28:49.302979 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dfd64f_0ef4_4176_9af6_a587e7e6c4cc.slice/crio-70bc07a0924ca367bf9fa5253f2a27f618d60fe91b5be3e02fd7ba3f39f08dc9 WatchSource:0}: Error finding container 70bc07a0924ca367bf9fa5253f2a27f618d60fe91b5be3e02fd7ba3f39f08dc9: Status 404 returned error can't find the container with id 70bc07a0924ca367bf9fa5253f2a27f618d60fe91b5be3e02fd7ba3f39f08dc9 Oct 30 20:28:49.305049 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.305022 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.305138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.305103 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.305182 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.305138 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:28:49.478162 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.478127 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" event={"ID":"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc","Type":"ContainerStarted","Data":"628852e74697a03fd4f94ddba364cac2182e41e7bc4417804877019a12d61335"} Oct 30 20:28:49.478392 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.478194 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" event={"ID":"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc","Type":"ContainerStarted","Data":"70bc07a0924ca367bf9fa5253f2a27f618d60fe91b5be3e02fd7ba3f39f08dc9"} Oct 30 20:28:49.480118 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.480085 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" event={"ID":"32aa4d2b-e4fa-4a68-883e-082788e51c95","Type":"ContainerStarted","Data":"ef0e29329da0a4fece5c2d51d088dc81c3daa3a5503053ace1b1c5acab6ddaae"} Oct 30 20:28:49.480248 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.480139 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" event={"ID":"32aa4d2b-e4fa-4a68-883e-082788e51c95","Type":"ContainerStarted","Data":"92eb246a38fec7b6a305d4847c56aadd995e8a59e139b31f6272a9b70ef4ecb3"} Oct 30 20:28:49.504724 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.504676 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podStartSLOduration=1.504661276 podStartE2EDuration="1.504661276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:49.500677805 +0000 UTC m=+3489.568513389" watchObservedRunningTime="2026-10-30 20:28:49.504661276 +0000 UTC m=+3489.572496844" Oct 30 20:28:49.525530 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:49.525178 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podStartSLOduration=1.525162803 podStartE2EDuration="1.525162803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:28:49.522929757 +0000 UTC m=+3489.590765326" watchObservedRunningTime="2026-10-30 20:28:49.525162803 +0000 UTC m=+3489.592998373" Oct 30 20:28:50.002674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.002642 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:50.007575 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.007549 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:50.010625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.010606 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:50.011827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.011806 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:28:50.011930 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.011848 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:28:50.486441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.486405 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:50.487235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:50.487216 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:28:51.014414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:51.014381 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:51.492543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:51.492506 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:51.493462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:51.493439 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:28:53.626213 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:53.626180 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:54.963801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:28:54.963763 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:28:59.377022 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:28:59.376978 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:29:29.377249 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:29:29.377211 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:29:43.338029 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.337988 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w"] Oct 30 20:29:43.341725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.341707 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.345427 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.345406 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kjn7n\"" Oct 30 20:29:43.357274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.357249 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w"] Oct 30 20:29:43.404044 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.403997 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76e6166d-79fd-4d5c-adde-24ab3b2b0400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.404277 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.404184 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84pnk\" (UniqueName: \"kubernetes.io/projected/76e6166d-79fd-4d5c-adde-24ab3b2b0400-kube-api-access-84pnk\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.505521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.505491 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84pnk\" (UniqueName: \"kubernetes.io/projected/76e6166d-79fd-4d5c-adde-24ab3b2b0400-kube-api-access-84pnk\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.505653 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.505558 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76e6166d-79fd-4d5c-adde-24ab3b2b0400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.505955 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.505934 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76e6166d-79fd-4d5c-adde-24ab3b2b0400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.518082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.518034 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84pnk\" (UniqueName: \"kubernetes.io/projected/76e6166d-79fd-4d5c-adde-24ab3b2b0400-kube-api-access-84pnk\") pod \"kuadrant-operator-controller-manager-678d87766f-pcl5w\" (UID: \"76e6166d-79fd-4d5c-adde-24ab3b2b0400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.655012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.654982 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:43.943744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:43.943719 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w"] Oct 30 20:29:43.945466 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:29:43.945438 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76e6166d_79fd_4d5c_adde_24ab3b2b0400.slice/crio-0b1344803214d04774d09f1d2dbab91473b54e147892a3962e9bc9c320aa1f89 WatchSource:0}: Error finding container 0b1344803214d04774d09f1d2dbab91473b54e147892a3962e9bc9c320aa1f89: Status 404 returned error can't find the container with id 0b1344803214d04774d09f1d2dbab91473b54e147892a3962e9bc9c320aa1f89 Oct 30 20:29:44.785482 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:44.785445 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" event={"ID":"76e6166d-79fd-4d5c-adde-24ab3b2b0400","Type":"ContainerStarted","Data":"0766f0bb44b20b852ab699e13f3f9a2e1724930ee613ac691bff2ecbbfca5f3b"} Oct 30 20:29:44.785909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:44.785501 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" event={"ID":"76e6166d-79fd-4d5c-adde-24ab3b2b0400","Type":"ContainerStarted","Data":"0b1344803214d04774d09f1d2dbab91473b54e147892a3962e9bc9c320aa1f89"} Oct 30 20:29:44.785909 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:44.785523 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:44.809581 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:44.809521 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" podStartSLOduration=1.809505845 podStartE2EDuration="1.809505845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:29:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:29:44.808011528 +0000 UTC m=+3544.875847097" watchObservedRunningTime="2026-10-30 20:29:44.809505845 +0000 UTC m=+3544.877341412" Oct 30 20:29:53.681214 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:53.681179 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:55.020572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:55.020540 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:29:55.792750 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:29:55.792724 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-pcl5w" Oct 30 20:29:59.375419 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:29:59.375382 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:30:04.762308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:04.762262 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:04.932275 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:04.932236 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:07.734083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.733591 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz"] Oct 30 20:30:07.763702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.763661 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz"] Oct 30 20:30:07.764152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.764049 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.770119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.769761 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:30:07.770313 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.770286 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:30:07.770522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.770462 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:30:07.770737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.770716 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:30:07.894853 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.894814 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfllz\" (UniqueName: \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.895035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.894885 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.895035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.894958 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.996042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.995948 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfllz\" (UniqueName: \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.996042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.996013 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.996260 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.996049 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.996786 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.996765 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:07.998521 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:07.998497 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:08.008172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:08.008141 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfllz\" (UniqueName: \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\") pod \"maas-api-revoke-sub-d576e68b-vbxnz\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:08.081432 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:08.081394 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:08.406091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:08.406040 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz"] Oct 30 20:30:08.409303 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:30:08.409276 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b23f730_181a_4c99_94c2_8ce7d1253f0d.slice/crio-53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46 WatchSource:0}: Error finding container 53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46: Status 404 returned error can't find the container with id 53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46 Oct 30 20:30:08.966081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:08.965495 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" event={"ID":"5b23f730-181a-4c99-94c2-8ce7d1253f0d","Type":"ContainerStarted","Data":"53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46"} Oct 30 20:30:09.975779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:09.975736 5733 generic.go:356] "Generic (PLEG): container finished" podID="5b23f730-181a-4c99-94c2-8ce7d1253f0d" containerID="e6d7d2171a16634ed3668640d69dd100044f596b32ee41823cd05dc0161fc007" exitCode=0 Oct 30 20:30:09.976239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:09.975832 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" event={"ID":"5b23f730-181a-4c99-94c2-8ce7d1253f0d","Type":"ContainerDied","Data":"e6d7d2171a16634ed3668640d69dd100044f596b32ee41823cd05dc0161fc007"} Oct 30 20:30:11.152184 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.152157 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:11.226945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.226903 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\") pod \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " Oct 30 20:30:11.227159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.226961 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\") pod \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " Oct 30 20:30:11.227159 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.226994 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\" (UniqueName: \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\") pod \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\" (UID: \"5b23f730-181a-4c99-94c2-8ce7d1253f0d\") " Oct 30 20:30:11.227431 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.227394 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca" pod "5b23f730-181a-4c99-94c2-8ce7d1253f0d" (UID: "5b23f730-181a-4c99-94c2-8ce7d1253f0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:30:11.229244 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.229214 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca" pod "5b23f730-181a-4c99-94c2-8ce7d1253f0d" (UID: "5b23f730-181a-4c99-94c2-8ce7d1253f0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:30:11.229359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.229332 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz" pod "5b23f730-181a-4c99-94c2-8ce7d1253f0d" (UID: "5b23f730-181a-4c99-94c2-8ce7d1253f0d"). InnerVolumeSpecName "kube-api-access-cfllz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:30:11.329106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.329040 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:11.329106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.329104 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b23f730-181a-4c99-94c2-8ce7d1253f0d-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:11.329327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.329124 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfllz\" (UniqueName: \"kubernetes.io/projected/5b23f730-181a-4c99-94c2-8ce7d1253f0d-kube-api-access-cfllz\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:30:11.707923 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.707884 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:11.990407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.990323 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" Oct 30 20:30:11.990407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.990354 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz" event={"ID":"5b23f730-181a-4c99-94c2-8ce7d1253f0d","Type":"ContainerDied","Data":"53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46"} Oct 30 20:30:11.990614 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:11.990419 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53bf3efba3369b302e82e40bd66a8cb656b4d2c2386f982864529510e8c12a46" Oct 30 20:30:12.984444 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:12.984405 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:14.004653 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.004621 5733 generic.go:356] "Generic (PLEG): container finished" podID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerID="ef0e29329da0a4fece5c2d51d088dc81c3daa3a5503053ace1b1c5acab6ddaae" exitCode=0 Oct 30 20:30:14.005205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.004695 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" event={"ID":"32aa4d2b-e4fa-4a68-883e-082788e51c95","Type":"ContainerDied","Data":"ef0e29329da0a4fece5c2d51d088dc81c3daa3a5503053ace1b1c5acab6ddaae"} Oct 30 20:30:14.005290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.005211 5733 scope.go:122] "RemoveContainer" containerID="ef0e29329da0a4fece5c2d51d088dc81c3daa3a5503053ace1b1c5acab6ddaae" Oct 30 20:30:14.006565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.006537 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:14.006638 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.006621 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:14.006677 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.006662 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:14.007428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.007408 5733 generic.go:356] "Generic (PLEG): container finished" podID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerID="628852e74697a03fd4f94ddba364cac2182e41e7bc4417804877019a12d61335" exitCode=0 Oct 30 20:30:14.007541 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.007484 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" event={"ID":"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc","Type":"ContainerDied","Data":"628852e74697a03fd4f94ddba364cac2182e41e7bc4417804877019a12d61335"} Oct 30 20:30:14.007831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.007814 5733 scope.go:122] "RemoveContainer" containerID="628852e74697a03fd4f94ddba364cac2182e41e7bc4417804877019a12d61335" Oct 30 20:30:14.010117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.010078 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:14.010266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.010241 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:14.010413 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:14.010392 5733 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 20:30:15.063334 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:15.063291 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" event={"ID":"f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc","Type":"ContainerStarted","Data":"f78f4a2d537132c482567476fed67be62fe23aff54323d8555407157aac76694"} Oct 30 20:30:15.066009 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:15.065978 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" event={"ID":"32aa4d2b-e4fa-4a68-883e-082788e51c95","Type":"ContainerStarted","Data":"e98a8c14a24dc23e6dc54e57f2af8aebe38b36f159f947909a7da8355229043e"} Oct 30 20:30:16.002552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.002512 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:30:16.003721 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.003684 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:16.003868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.003757 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:16.030082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.022894 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:30:16.030082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.023046 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:16.030082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:16.023132 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:17.002539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:17.002502 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:17.002935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:17.002559 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:17.016198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:17.016159 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:17.016475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:17.016443 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:18.003181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:18.003145 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:18.003615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:18.003197 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:18.018082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:18.014908 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:18.018082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:18.014962 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:19.003404 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:19.003367 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:19.003806 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:19.003428 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:19.018341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:19.018191 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:19.018341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:19.018266 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:20.002783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:20.002743 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:20.002971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:20.002804 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:20.015080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:20.014444 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:20.015080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:20.014501 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:20.486966 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:20.486921 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:30:21.002962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:21.002921 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body= Oct 30 20:30:21.003154 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:21.002997 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" Oct 30 20:30:21.010508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:21.010477 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body= Oct 30 20:30:21.010658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:21.010530 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" Oct 30 20:30:21.492737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:21.492683 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:30:23.003755 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:23.003716 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:23.004339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:23.003816 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:23.014996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:23.014962 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:23.015298 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:23.015266 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:24.049969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:24.049142 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:24.049969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:24.049207 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:24.074637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:24.074152 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:24.074637 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:24.074216 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:25.090227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:25.090187 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:25.090734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:25.090267 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:25.116304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:25.116246 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:25.116510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:25.116312 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:26.091366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:26.091308 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:26.091933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:26.091380 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:26.117087 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:26.117021 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:26.117278 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:26.117104 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:27.133308 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:27.133256 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-qr62s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:27.134005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:27.133954 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" podUID="32aa4d2b-e4fa-4a68-883e-082788e51c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:27.158234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:27.158191 5733 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-8fcjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:30:27.158376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:27.158251 5733 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" podUID="f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:30:28.077548 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:28.077516 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:30:28.078627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:28.078603 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-8fcjn" Oct 30 20:30:28.101400 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:28.101371 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:30:28.103239 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:28.103208 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-qr62s" Oct 30 20:30:29.376554 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:30:29.376516 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:30:38.590782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:38.590731 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.068268 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:39.068225 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:39.243076 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:39.243027 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:41.015366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:41.015338 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:30:41.026369 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:41.026345 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:30:42.949993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:42.949942 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:43.135255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:43.135220 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.229153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.229113 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.319988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.319955 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:30:44.320417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.320404 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5b23f730-181a-4c99-94c2-8ce7d1253f0d" containerName="revoke-keys" Oct 30 20:30:44.320466 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.320418 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b23f730-181a-4c99-94c2-8ce7d1253f0d" containerName="revoke-keys" Oct 30 20:30:44.320508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.320489 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b23f730-181a-4c99-94c2-8ce7d1253f0d" containerName="revoke-keys" Oct 30 20:30:44.323792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.323774 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.339105 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.339077 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-de61d2\"" Oct 30 20:30:44.341603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.341582 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e3f80981-dockercfg-bdw5f\"" Oct 30 20:30:44.341935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.341910 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:30:44.345596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.345574 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:30:44.345707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.345683 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.359477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.359446 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:30:44.411613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.411569 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qlf\" (UniqueName: \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.411826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.411670 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.411826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.411722 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvtc\" (UniqueName: \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.411826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.411799 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.413694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.413654 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.513317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.513227 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvtc\" (UniqueName: \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.513317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.513292 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.513543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.513351 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qlf\" (UniqueName: \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.513543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.513414 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.514100 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.514043 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.514187 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.514167 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.528139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.528112 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvtc\" (UniqueName: \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\") pod \"payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.528263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.528156 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qlf\" (UniqueName: \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\") pod \"payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.637462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.637421 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:44.664263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.664234 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:44.685696 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.685664 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.833347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.833257 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:44.922920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.922892 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:30:44.925209 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:30:44.925180 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08453361_3da7_4556_99a3_a0d04ef43e52.slice/crio-c297dd2c889138864faadea9623e170db009e9daa0e4ee26bd6ed5caed50fd7d WatchSource:0}: Error finding container c297dd2c889138864faadea9623e170db009e9daa0e4ee26bd6ed5caed50fd7d: Status 404 returned error can't find the container with id c297dd2c889138864faadea9623e170db009e9daa0e4ee26bd6ed5caed50fd7d Oct 30 20:30:44.953900 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:44.953883 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:30:45.261149 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.261111 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" event={"ID":"08453361-3da7-4556-99a3-a0d04ef43e52","Type":"ContainerStarted","Data":"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3"} Oct 30 20:30:45.261619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.261172 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" event={"ID":"08453361-3da7-4556-99a3-a0d04ef43e52","Type":"ContainerStarted","Data":"c297dd2c889138864faadea9623e170db009e9daa0e4ee26bd6ed5caed50fd7d"} Oct 30 20:30:45.261619 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.261197 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:45.262780 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.262747 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" event={"ID":"e3680ecb-8e12-4dc1-a24a-6148dd78f176","Type":"ContainerStarted","Data":"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f"} Oct 30 20:30:45.262877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.262799 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" event={"ID":"e3680ecb-8e12-4dc1-a24a-6148dd78f176","Type":"ContainerStarted","Data":"c6d7cb2ff13f5a9ce02d9b6d8d7a0b6e43706bcd943219586915661add2048c2"} Oct 30 20:30:45.262919 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.262879 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:45.281606 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.281554 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" podStartSLOduration=1.281535085 podStartE2EDuration="1.281535085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:45.279997574 +0000 UTC m=+3605.347833142" watchObservedRunningTime="2026-10-30 20:30:45.281535085 +0000 UTC m=+3605.349370655" Oct 30 20:30:45.300805 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.300751 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" podStartSLOduration=1.300731949 podStartE2EDuration="1.300731949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:30:45.297219716 +0000 UTC m=+3605.365055283" watchObservedRunningTime="2026-10-30 20:30:45.300731949 +0000 UTC m=+3605.368567516" Oct 30 20:30:45.304939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:45.304900 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:46.102496 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:46.102456 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:47.018220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:47.014943 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:47.808505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:47.808467 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:48.589818 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:48.589776 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:49.424208 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:49.424159 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:50.402325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:50.402290 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:51.195537 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:51.195415 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:51.340227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:51.340186 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:52.002130 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:52.002094 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:52.796762 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:52.796711 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:53.629290 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:53.629216 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:53.824192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:53.824156 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:54.488773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:54.488738 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:55.078349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:55.078316 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:55.314716 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:55.314679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:56.225727 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:56.225690 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:56.273644 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:56.273617 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:30:56.273793 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:56.273690 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:30:57.442485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:57.442452 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:57.799917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:57.799814 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.024550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:58.024500 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.643669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:58.643637 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.812217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:58.812093 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:58.953691 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:58.953623 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:30:59.376350 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:30:59.376314 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:30:59.703361 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:30:59.703242 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.078180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:00.078049 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:00.701545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:00.701512 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:01.511584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:01.511519 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:02.406176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:02.406144 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:03.192805 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:03.192769 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:04.104875 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:04.104839 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:04.970702 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:04.970648 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:05.801939 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:05.801901 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:06.695235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:06.695201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:07.499302 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:07.499265 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:08.296252 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:08.296209 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:09.092150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:09.092111 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:09.901758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:09.901726 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:10.713835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:10.713805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:11.489172 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:11.489125 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:12.293088 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:12.293040 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.092418 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:13.092373 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:13.896764 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:13.896713 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:14.705006 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:14.704972 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:15.493823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:15.493769 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:16.348415 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:16.348381 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:17.313003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:17.312970 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:18.138589 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:18.138553 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:19.121353 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:19.121321 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:19.899744 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:19.899704 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:20.793070 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:20.793030 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:21.602180 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:21.602128 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:22.403546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:22.403512 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:23.197616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:23.197587 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:24.008835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:24.008803 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:24.779837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:24.779783 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:25.586221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:25.586189 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:26.397934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:26.397872 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:27.208601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:27.208562 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:28.023726 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:28.023694 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:28.810158 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:28.810119 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:29.377292 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:31:29.377254 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:31:29.666692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:29.666619 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:30.426546 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.426513 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:30.454463 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.454409 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf"] Oct 30 20:31:30.458500 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.458482 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.467557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.466918 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:31:30.467557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.467092 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:31:30.467557 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.467434 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:31:30.467905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.467883 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:31:30.471715 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.471690 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf"] Oct 30 20:31:30.579960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.579924 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.580188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.580087 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtj4\" (UniqueName: \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.580188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.580126 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.681505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.681412 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtj4\" (UniqueName: \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.681505 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.681482 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.681739 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.681528 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.682279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.682261 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.684016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.683993 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.694490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.694465 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtj4\" (UniqueName: \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\") pod \"maas-api-revoke-sub-11a28650-vrmqf\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:30.774794 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:30.774758 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:31.067006 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:31.066975 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf"] Oct 30 20:31:31.068430 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:31:31.068399 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf72d073_8db1_48bf_82db_73a8fea145a0.slice/crio-605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318 WatchSource:0}: Error finding container 605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318: Status 404 returned error can't find the container with id 605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318 Oct 30 20:31:31.290931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:31.290897 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:31.529969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:31.529924 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" event={"ID":"cf72d073-8db1-48bf-82db-73a8fea145a0","Type":"ContainerStarted","Data":"0d25fd3c1c87d45d6dc480348a34348705a0eae1e4b77496ca3c1a03cdb370f2"} Oct 30 20:31:31.530509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:31.530000 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" event={"ID":"cf72d073-8db1-48bf-82db-73a8fea145a0","Type":"ContainerStarted","Data":"605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318"} Oct 30 20:31:31.551770 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:31.551713 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" podStartSLOduration=1.551699001 podStartE2EDuration="1.551699001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:31.547937053 +0000 UTC m=+3651.615772621" watchObservedRunningTime="2026-10-30 20:31:31.551699001 +0000 UTC m=+3651.619534569" Oct 30 20:31:32.275117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:32.274998 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:32.538110 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:32.538006 5733 generic.go:356] "Generic (PLEG): container finished" podID="cf72d073-8db1-48bf-82db-73a8fea145a0" containerID="0d25fd3c1c87d45d6dc480348a34348705a0eae1e4b77496ca3c1a03cdb370f2" exitCode=0 Oct 30 20:31:32.538549 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:32.538102 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" event={"ID":"cf72d073-8db1-48bf-82db-73a8fea145a0","Type":"ContainerDied","Data":"0d25fd3c1c87d45d6dc480348a34348705a0eae1e4b77496ca3c1a03cdb370f2"} Oct 30 20:31:33.079669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.079634 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:33.676641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.676609 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:33.816963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.816880 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\" (UniqueName: \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\") pod \"cf72d073-8db1-48bf-82db-73a8fea145a0\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " Oct 30 20:31:33.817168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.816991 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\") pod \"cf72d073-8db1-48bf-82db-73a8fea145a0\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " Oct 30 20:31:33.817168 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.817053 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\") pod \"cf72d073-8db1-48bf-82db-73a8fea145a0\" (UID: \"cf72d073-8db1-48bf-82db-73a8fea145a0\") " Oct 30 20:31:33.817765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.817720 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca" pod "cf72d073-8db1-48bf-82db-73a8fea145a0" (UID: "cf72d073-8db1-48bf-82db-73a8fea145a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:33.819323 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.819285 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4" pod "cf72d073-8db1-48bf-82db-73a8fea145a0" (UID: "cf72d073-8db1-48bf-82db-73a8fea145a0"). InnerVolumeSpecName "kube-api-access-4dtj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:33.819906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.819876 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca" pod "cf72d073-8db1-48bf-82db-73a8fea145a0" (UID: "cf72d073-8db1-48bf-82db-73a8fea145a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:33.879005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.878974 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:33.918103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.918068 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:33.918103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.918098 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf72d073-8db1-48bf-82db-73a8fea145a0-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:33.918103 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:33.918111 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtj4\" (UniqueName: \"kubernetes.io/projected/cf72d073-8db1-48bf-82db-73a8fea145a0-kube-api-access-4dtj4\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:34.551650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:34.551608 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" event={"ID":"cf72d073-8db1-48bf-82db-73a8fea145a0","Type":"ContainerDied","Data":"605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318"} Oct 30 20:31:34.551650 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:34.551651 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf" Oct 30 20:31:34.551917 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:34.551670 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605acbeb0b6a2cb636aa73f13d8dbb0ba839ff94bebe8d555e98e709b7882318" Oct 30 20:31:34.688484 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:34.688452 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:35.495692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:35.495652 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:36.289928 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:36.289805 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:37.093841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:37.093794 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:37.999742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:37.999708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:38.815266 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:38.815235 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:39.596339 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:39.596303 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:40.512723 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:40.512684 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:41.287545 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.287503 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:41.440797 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.440760 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq"] Oct 30 20:31:41.441286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.441265 5733 cpu_manager.go:414] "containerMap: removing container" podUID="cf72d073-8db1-48bf-82db-73a8fea145a0" containerName="revoke-keys" Oct 30 20:31:41.441414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.441288 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf72d073-8db1-48bf-82db-73a8fea145a0" containerName="revoke-keys" Oct 30 20:31:41.441414 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.441409 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf72d073-8db1-48bf-82db-73a8fea145a0" containerName="revoke-keys" Oct 30 20:31:41.444935 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.444918 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.452385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.452231 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:31:41.452385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.452347 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:31:41.452385 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.452356 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:31:41.452568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.452294 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:31:41.460938 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.460913 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq"] Oct 30 20:31:41.462417 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.462388 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:41.597861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.597836 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsr4r\" (UniqueName: \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.598017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.597966 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.598017 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.597996 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.699502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.699467 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsr4r\" (UniqueName: \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.699674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.699550 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.699674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.699570 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.700238 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.700215 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.701920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.701898 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.712030 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.712007 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsr4r\" (UniqueName: \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\") pod \"maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:41.758679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:41.758649 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:42.062176 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:42.062149 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq"] Oct 30 20:31:42.064129 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:31:42.064096 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0b24be_9eb0_4c98_8006_2e999395e8e3.slice/crio-995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe WatchSource:0}: Error finding container 995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe: Status 404 returned error can't find the container with id 995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe Oct 30 20:31:42.483235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:42.483205 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:42.601249 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:42.601215 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" event={"ID":"8d0b24be-9eb0-4c98-8006-2e999395e8e3","Type":"ContainerStarted","Data":"74495bfa43eea8d96c653609e3acc30ff1af80ea02a41fc404420aaa9b016662"} Oct 30 20:31:42.601420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:42.601277 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" event={"ID":"8d0b24be-9eb0-4c98-8006-2e999395e8e3","Type":"ContainerStarted","Data":"995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe"} Oct 30 20:31:42.624622 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:42.624573 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" podStartSLOduration=1.624559635 podStartE2EDuration="1.624559635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:31:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:31:42.62159642 +0000 UTC m=+3662.689431989" watchObservedRunningTime="2026-10-30 20:31:42.624559635 +0000 UTC m=+3662.692395203" Oct 30 20:31:43.575163 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:43.575127 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:43.607791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:43.607750 5733 generic.go:356] "Generic (PLEG): container finished" podID="8d0b24be-9eb0-4c98-8006-2e999395e8e3" containerID="74495bfa43eea8d96c653609e3acc30ff1af80ea02a41fc404420aaa9b016662" exitCode=0 Oct 30 20:31:43.607791 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:43.607779 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" event={"ID":"8d0b24be-9eb0-4c98-8006-2e999395e8e3","Type":"ContainerDied","Data":"74495bfa43eea8d96c653609e3acc30ff1af80ea02a41fc404420aaa9b016662"} Oct 30 20:31:44.408906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.408869 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:44.761037 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.761009 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:44.933673 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.933578 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\") pod \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " Oct 30 20:31:44.933850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.933701 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\") pod \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " Oct 30 20:31:44.933850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.933813 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\" (UniqueName: \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\") pod \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\" (UID: \"8d0b24be-9eb0-4c98-8006-2e999395e8e3\") " Oct 30 20:31:44.934186 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.934146 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca" pod "8d0b24be-9eb0-4c98-8006-2e999395e8e3" (UID: "8d0b24be-9eb0-4c98-8006-2e999395e8e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:31:44.935931 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.935902 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r" pod "8d0b24be-9eb0-4c98-8006-2e999395e8e3" (UID: "8d0b24be-9eb0-4c98-8006-2e999395e8e3"). InnerVolumeSpecName "kube-api-access-tsr4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:31:44.936255 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:44.936232 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca" pod "8d0b24be-9eb0-4c98-8006-2e999395e8e3" (UID: "8d0b24be-9eb0-4c98-8006-2e999395e8e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:31:45.035518 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.035465 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:45.035518 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.035513 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d0b24be-9eb0-4c98-8006-2e999395e8e3-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:45.035518 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.035528 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsr4r\" (UniqueName: \"kubernetes.io/projected/8d0b24be-9eb0-4c98-8006-2e999395e8e3-kube-api-access-tsr4r\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:31:45.315640 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.315548 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:45.623034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.623005 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" Oct 30 20:31:45.623230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.623011 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq" event={"ID":"8d0b24be-9eb0-4c98-8006-2e999395e8e3","Type":"ContainerDied","Data":"995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe"} Oct 30 20:31:45.623230 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:45.623134 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995252ea524664c1fa02e638fb2289922b06b55f9592345d7951f0dba6bbeabe" Oct 30 20:31:46.173216 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:46.173181 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:46.403756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:46.403724 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:46.993226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:46.993188 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:47.786735 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:47.786700 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:48.591040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:48.591001 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:49.507814 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:49.507765 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:50.298993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:50.298923 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.088861 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:51.088824 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.478528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:51.478492 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:31:51.499980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:51.499938 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:31:51.670045 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:51.670011 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:51.951394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:51.951359 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:52.138503 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:52.138461 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:52.809551 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:52.809441 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq"] Oct 30 20:31:52.812846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:52.812816 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-de61d2-tgwtq"] Oct 30 20:31:52.833282 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:52.833253 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:52.974469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:52.974438 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:53.480960 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:53.480926 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:53.797286 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:53.797183 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:54.050531 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:54.050441 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0b24be-9eb0-4c98-8006-2e999395e8e3" path="/var/lib/kubelet/pods/8d0b24be-9eb0-4c98-8006-2e999395e8e3/volumes" Oct 30 20:31:55.129964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:55.129923 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:56.479421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:56.479368 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" containerID="cri-o://456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f" gracePeriod=25 Oct 30 20:31:56.501026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:56.500988 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" containerID="cri-o://7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3" gracePeriod=25 Oct 30 20:31:59.377421 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:31:59.377382 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:31:59.760363 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:59.760253 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:31:59.908672 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:31:59.908620 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:00.086311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:00.086205 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:05.167433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:05.167392 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:06.271573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:06.271530 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:32:06.271969 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:06.271530 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:32:16.271654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.271610 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:32:16.271654 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.271631 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:32:16.754921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.754893 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:32:16.788914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.788884 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:32:16.836596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.836515 5733 generic.go:356] "Generic (PLEG): container finished" podID="08453361-3da7-4556-99a3-a0d04ef43e52" containerID="7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3" exitCode=0 Oct 30 20:32:16.836596 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.836585 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" Oct 30 20:32:16.836843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.836606 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" event={"ID":"08453361-3da7-4556-99a3-a0d04ef43e52","Type":"ContainerDied","Data":"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3"} Oct 30 20:32:16.836843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.836673 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5" event={"ID":"08453361-3da7-4556-99a3-a0d04ef43e52","Type":"ContainerDied","Data":"c297dd2c889138864faadea9623e170db009e9daa0e4ee26bd6ed5caed50fd7d"} Oct 30 20:32:16.836843 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.836705 5733 scope.go:122] "RemoveContainer" containerID="7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3" Oct 30 20:32:16.838211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.838190 5733 generic.go:356] "Generic (PLEG): container finished" podID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerID="456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f" exitCode=0 Oct 30 20:32:16.838330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.838241 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" event={"ID":"e3680ecb-8e12-4dc1-a24a-6148dd78f176","Type":"ContainerDied","Data":"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f"} Oct 30 20:32:16.838330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.838272 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" event={"ID":"e3680ecb-8e12-4dc1-a24a-6148dd78f176","Type":"ContainerDied","Data":"c6d7cb2ff13f5a9ce02d9b6d8d7a0b6e43706bcd943219586915661add2048c2"} Oct 30 20:32:16.838330 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.838318 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz" Oct 30 20:32:16.846364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.846342 5733 scope.go:122] "RemoveContainer" containerID="7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3" Oct 30 20:32:16.846651 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:32:16.846626 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3\": container with ID starting with 7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3 not found: ID does not exist" containerID="7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3" Oct 30 20:32:16.846743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.846657 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3"} err="failed to get container status \"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3\": rpc error: code = NotFound desc = could not find container \"7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3\": container with ID starting with 7f70de41d0bd9ec3dd87a067a8cb62245a2ba87e38aefa99a23079946a6912b3 not found: ID does not exist" Oct 30 20:32:16.846743 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.846672 5733 scope.go:122] "RemoveContainer" containerID="456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f" Oct 30 20:32:16.855423 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.855406 5733 scope.go:122] "RemoveContainer" containerID="456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f" Oct 30 20:32:16.855700 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:32:16.855680 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f\": container with ID starting with 456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f not found: ID does not exist" containerID="456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f" Oct 30 20:32:16.855759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.855707 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f"} err="failed to get container status \"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f\": rpc error: code = NotFound desc = could not find container \"456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f\": container with ID starting with 456661c605ac2b74c9f3450f4a15101dcfa44839bca7dbe084074204f0e00d6f not found: ID does not exist" Oct 30 20:32:16.887720 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.887693 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\" (UniqueName: \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\") pod \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " Oct 30 20:32:16.887819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.887725 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\" (UniqueName: \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\") pod \"08453361-3da7-4556-99a3-a0d04ef43e52\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " Oct 30 20:32:16.887945 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.887928 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\") pod \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\" (UID: \"e3680ecb-8e12-4dc1-a24a-6148dd78f176\") " Oct 30 20:32:16.888036 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.887965 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\") pod \"08453361-3da7-4556-99a3-a0d04ef43e52\" (UID: \"08453361-3da7-4556-99a3-a0d04ef43e52\") " Oct 30 20:32:16.888345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.888326 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume" pod "e3680ecb-8e12-4dc1-a24a-6148dd78f176" (UID: "e3680ecb-8e12-4dc1-a24a-6148dd78f176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:16.888450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.888428 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume" pod "08453361-3da7-4556-99a3-a0d04ef43e52" (UID: "08453361-3da7-4556-99a3-a0d04ef43e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:16.889767 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.889747 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc" pod "e3680ecb-8e12-4dc1-a24a-6148dd78f176" (UID: "e3680ecb-8e12-4dc1-a24a-6148dd78f176"). InnerVolumeSpecName "kube-api-access-fnvtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:16.889825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.889802 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf" pod "08453361-3da7-4556-99a3-a0d04ef43e52" (UID: "08453361-3da7-4556-99a3-a0d04ef43e52"). InnerVolumeSpecName "kube-api-access-n5qlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:16.989655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.989627 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3680ecb-8e12-4dc1-a24a-6148dd78f176-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.989655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.989651 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08453361-3da7-4556-99a3-a0d04ef43e52-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.989811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.989661 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnvtc\" (UniqueName: \"kubernetes.io/projected/e3680ecb-8e12-4dc1-a24a-6148dd78f176-kube-api-access-fnvtc\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:16.989811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:16.989670 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qlf\" (UniqueName: \"kubernetes.io/projected/08453361-3da7-4556-99a3-a0d04ef43e52-kube-api-access-n5qlf\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:17.187906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:17.187865 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:32:17.193143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:17.193120 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-de61d2-6b4d5kkwz5"] Oct 30 20:32:17.221475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:17.221452 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:32:17.228655 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:17.228630 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-de61d2-7989548449d8lz"] Oct 30 20:32:18.048807 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:18.048773 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" path="/var/lib/kubelet/pods/08453361-3da7-4556-99a3-a0d04ef43e52/volumes" Oct 30 20:32:18.049217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:18.049174 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" path="/var/lib/kubelet/pods/e3680ecb-8e12-4dc1-a24a-6148dd78f176/volumes" Oct 30 20:32:21.810527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810440 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:32:21.810920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810905 5733 cpu_manager.go:414] "containerMap: removing container" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" Oct 30 20:32:21.810963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810922 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" Oct 30 20:32:21.810963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810939 5733 cpu_manager.go:414] "containerMap: removing container" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" Oct 30 20:32:21.810963 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810944 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" Oct 30 20:32:21.811086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810965 5733 cpu_manager.go:414] "containerMap: removing container" podUID="8d0b24be-9eb0-4c98-8006-2e999395e8e3" containerName="revoke-keys" Oct 30 20:32:21.811086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.810971 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0b24be-9eb0-4c98-8006-2e999395e8e3" containerName="revoke-keys" Oct 30 20:32:21.811086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.811032 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3680ecb-8e12-4dc1-a24a-6148dd78f176" containerName="payload-processing" Oct 30 20:32:21.811086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.811045 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0b24be-9eb0-4c98-8006-2e999395e8e3" containerName="revoke-keys" Oct 30 20:32:21.811086 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.811071 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="08453361-3da7-4556-99a3-a0d04ef43e52" containerName="payload-pre-processing" Oct 30 20:32:21.815632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.815610 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.820694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.820667 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-381a9150-kserve-self-signed-certs\"" Oct 30 20:32:21.829511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.829486 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:32:21.910583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.910546 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:21.940513 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940476 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6ql\" (UniqueName: \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.940717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940542 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.940717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940631 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.940717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940677 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.940878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940738 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:21.940878 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:21.940755 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.041848 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.041805 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.041873 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.041913 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.041937 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042111 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.042008 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6ql\" (UniqueName: \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.042108 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042377 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.042313 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.042587 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.042769 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.042656 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.044540 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.044517 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.044674 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.044599 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.054462 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.054432 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6ql\" (UniqueName: \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\") pod \"e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.128628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.128593 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:22.413080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.413031 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:32:22.415310 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:32:22.415281 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb50fef2_380e_4357_af32_6dcb3ec36f32.slice/crio-a7df5975e86742f0bcdfeb65f736f1d13d227d0080aaa642dd3f8126581d641c WatchSource:0}: Error finding container a7df5975e86742f0bcdfeb65f736f1d13d227d0080aaa642dd3f8126581d641c: Status 404 returned error can't find the container with id a7df5975e86742f0bcdfeb65f736f1d13d227d0080aaa642dd3f8126581d641c Oct 30 20:32:22.880676 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.880639 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" event={"ID":"fb50fef2-380e-4357-af32-6dcb3ec36f32","Type":"ContainerStarted","Data":"ee3c574c192368417035048d601a4875afe8b025ebd33839c86dcd2c92b8c816"} Oct 30 20:32:22.881092 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.880691 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" event={"ID":"fb50fef2-380e-4357-af32-6dcb3ec36f32","Type":"ContainerStarted","Data":"a7df5975e86742f0bcdfeb65f736f1d13d227d0080aaa642dd3f8126581d641c"} Oct 30 20:32:22.910372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:22.910322 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" podStartSLOduration=1.9103082059999998 podStartE2EDuration="1.910308206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:22.908794716 +0000 UTC m=+3702.976630281" watchObservedRunningTime="2026-10-30 20:32:22.910308206 +0000 UTC m=+3702.978143779" Oct 30 20:32:23.129513 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:23.129476 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:23.399693 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:23.399573 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:24.902446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:24.902406 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:29.377303 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:32:29.377263 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:32:30.829209 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:30.829174 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:31.019437 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:31.019388 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:32.128989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:32.128962 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:32.141517 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:32.141493 5733 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:32.954896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:32.954867 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:32:38.050841 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.050811 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk"] Oct 30 20:32:38.055446 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.055429 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.060396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.060371 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:32:38.060573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.060463 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:32:38.060573 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.060529 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:32:38.060747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.060666 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:32:38.063320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.063296 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk"] Oct 30 20:32:38.130774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.130738 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.130944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.130883 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnmb8\" (UniqueName: \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.130944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.130916 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.231477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.231440 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnmb8\" (UniqueName: \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.231477 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.231482 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.231697 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.231540 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.232271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.232251 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.234000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.233979 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.244034 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.244008 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnmb8\" (UniqueName: \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\") pod \"maas-api-revoke-sub-452b6d7b-4wnhk\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.373246 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.373210 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:38.667815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.667783 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk"] Oct 30 20:32:38.669148 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:32:38.669115 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ffe5fa_a599_46c0_a5b3_449df8067a1d.slice/crio-22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b WatchSource:0}: Error finding container 22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b: Status 404 returned error can't find the container with id 22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b Oct 30 20:32:38.977989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.977953 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" event={"ID":"33ffe5fa-a599-46c0-a5b3-449df8067a1d","Type":"ContainerStarted","Data":"549d744f38e13ca177c9055929eb71a1a3f8048e4b499f2112b82c6ceada484d"} Oct 30 20:32:38.978164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:38.978005 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" event={"ID":"33ffe5fa-a599-46c0-a5b3-449df8067a1d","Type":"ContainerStarted","Data":"22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b"} Oct 30 20:32:39.006052 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:39.006001 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" podStartSLOduration=1.005988025 podStartE2EDuration="1.005988025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:39.002920536 +0000 UTC m=+3719.070756103" watchObservedRunningTime="2026-10-30 20:32:39.005988025 +0000 UTC m=+3719.073823593" Oct 30 20:32:39.989281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:39.989228 5733 generic.go:356] "Generic (PLEG): container finished" podID="33ffe5fa-a599-46c0-a5b3-449df8067a1d" containerID="549d744f38e13ca177c9055929eb71a1a3f8048e4b499f2112b82c6ceada484d" exitCode=0 Oct 30 20:32:39.989825 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:39.989400 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" event={"ID":"33ffe5fa-a599-46c0-a5b3-449df8067a1d","Type":"ContainerDied","Data":"549d744f38e13ca177c9055929eb71a1a3f8048e4b499f2112b82c6ceada484d"} Oct 30 20:32:41.582591 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.582235 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:41.610778 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.610674 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\" (UniqueName: \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\") pod \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " Oct 30 20:32:41.610981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.610810 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\") pod \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " Oct 30 20:32:41.610981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.610891 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\") pod \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\" (UID: \"33ffe5fa-a599-46c0-a5b3-449df8067a1d\") " Oct 30 20:32:41.614568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.614151 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca" pod "33ffe5fa-a599-46c0-a5b3-449df8067a1d" (UID: "33ffe5fa-a599-46c0-a5b3-449df8067a1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:41.626912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.626871 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca" pod "33ffe5fa-a599-46c0-a5b3-449df8067a1d" (UID: "33ffe5fa-a599-46c0-a5b3-449df8067a1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:41.651444 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.645521 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8" pod "33ffe5fa-a599-46c0-a5b3-449df8067a1d" (UID: "33ffe5fa-a599-46c0-a5b3-449df8067a1d"). InnerVolumeSpecName "kube-api-access-mnmb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:41.734114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.711907 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:41.734114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.711957 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33ffe5fa-a599-46c0-a5b3-449df8067a1d-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:41.734114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:41.711975 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnmb8\" (UniqueName: \"kubernetes.io/projected/33ffe5fa-a599-46c0-a5b3-449df8067a1d-kube-api-access-mnmb8\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:42.012407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:42.012365 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" event={"ID":"33ffe5fa-a599-46c0-a5b3-449df8067a1d","Type":"ContainerDied","Data":"22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b"} Oct 30 20:32:42.012694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:42.012674 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22946271b8d3f7905c2edc5cb86e39a9e5d2f604bd32eb3eb954e587c5e76c4b" Oct 30 20:32:42.012880 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:42.012865 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk" Oct 30 20:32:49.068235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:49.068198 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:51.788099 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:51.788026 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:32:51.868081 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:51.868014 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:53.993921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:53.993851 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:54.834758 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.834716 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t"] Oct 30 20:32:54.835271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.835251 5733 cpu_manager.go:414] "containerMap: removing container" podUID="33ffe5fa-a599-46c0-a5b3-449df8067a1d" containerName="revoke-keys" Oct 30 20:32:54.835341 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.835272 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ffe5fa-a599-46c0-a5b3-449df8067a1d" containerName="revoke-keys" Oct 30 20:32:54.835382 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.835365 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ffe5fa-a599-46c0-a5b3-449df8067a1d" containerName="revoke-keys" Oct 30 20:32:54.838424 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.838406 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.848243 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.848219 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:32:54.849098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.849051 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:32:54.853734 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.853708 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t"] Oct 30 20:32:54.895344 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.895281 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.895492 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.895462 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk59j\" (UniqueName: \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.895553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.895531 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.934469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.934434 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:32:54.934681 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.934501 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6zmll\"" Oct 30 20:32:54.996819 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.996764 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk59j\" (UniqueName: \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.997391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.996837 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.997391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.997004 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.997609 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.997583 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:54.999664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:54.999632 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:55.009242 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:55.009218 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk59j\" (UniqueName: \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\") pod \"maas-api-revoke-sub-3e31a506-krf2t\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:55.152699 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:55.152661 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:55.192657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:55.192611 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:32:55.461550 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:55.461515 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t"] Oct 30 20:32:55.464660 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:32:55.464572 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d2ad8b_f5a0_4829_8003_600a14baaa6d.slice/crio-071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2 WatchSource:0}: Error finding container 071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2: Status 404 returned error can't find the container with id 071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2 Oct 30 20:32:56.154679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:56.153587 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" event={"ID":"08d2ad8b-f5a0-4829-8003-600a14baaa6d","Type":"ContainerStarted","Data":"c10f3ae91ebf7dd4aa52364fee8aea4a816b13882315fe7e3abe5fb45c3762dd"} Oct 30 20:32:56.154679 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:56.153669 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" event={"ID":"08d2ad8b-f5a0-4829-8003-600a14baaa6d","Type":"ContainerStarted","Data":"071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2"} Oct 30 20:32:56.184592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:56.184528 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" podStartSLOduration=2.184507988 podStartE2EDuration="2.184507988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:32:56.174400883 +0000 UTC m=+3736.242236451" watchObservedRunningTime="2026-10-30 20:32:56.184507988 +0000 UTC m=+3736.252343556" Oct 30 20:32:57.171914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:57.171880 5733 generic.go:356] "Generic (PLEG): container finished" podID="08d2ad8b-f5a0-4829-8003-600a14baaa6d" containerID="c10f3ae91ebf7dd4aa52364fee8aea4a816b13882315fe7e3abe5fb45c3762dd" exitCode=0 Oct 30 20:32:57.172668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:57.172640 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" event={"ID":"08d2ad8b-f5a0-4829-8003-600a14baaa6d","Type":"ContainerDied","Data":"c10f3ae91ebf7dd4aa52364fee8aea4a816b13882315fe7e3abe5fb45c3762dd"} Oct 30 20:32:58.321253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.321231 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:58.352227 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.352190 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\") pod \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " Oct 30 20:32:58.352783 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.352752 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca" pod "08d2ad8b-f5a0-4829-8003-600a14baaa6d" (UID: "08d2ad8b-f5a0-4829-8003-600a14baaa6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:32:58.352904 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.352426 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\") pod \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " Oct 30 20:32:58.352977 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.352905 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\" (UniqueName: \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\") pod \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\" (UID: \"08d2ad8b-f5a0-4829-8003-600a14baaa6d\") " Oct 30 20:32:58.353428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.353374 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-openshift-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:58.354951 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.354919 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca" pod "08d2ad8b-f5a0-4829-8003-600a14baaa6d" (UID: "08d2ad8b-f5a0-4829-8003-600a14baaa6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:32:58.355580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.355554 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j" pod "08d2ad8b-f5a0-4829-8003-600a14baaa6d" (UID: "08d2ad8b-f5a0-4829-8003-600a14baaa6d"). InnerVolumeSpecName "kube-api-access-zk59j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:32:58.454337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.454253 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d2ad8b-f5a0-4829-8003-600a14baaa6d-maas-api-xks-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:58.454337 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:58.454284 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk59j\" (UniqueName: \"kubernetes.io/projected/08d2ad8b-f5a0-4829-8003-600a14baaa6d-kube-api-access-zk59j\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:32:59.186539 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:59.186508 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" Oct 30 20:32:59.186714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:59.186537 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t" event={"ID":"08d2ad8b-f5a0-4829-8003-600a14baaa6d","Type":"ContainerDied","Data":"071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2"} Oct 30 20:32:59.186714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:32:59.186594 5733 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="071693ba3fff25cf35a509ca749f400f31a24c1836fe0970288784e3b564f1c2" Oct 30 20:32:59.376636 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:32:59.376591 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:33:06.845399 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:06.845307 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerName="main" containerID="cri-o://ee3c574c192368417035048d601a4875afe8b025ebd33839c86dcd2c92b8c816" gracePeriod=45 Oct 30 20:33:06.945294 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:06.945237 5733 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerName="main" probeResult="failure" output="Get \"https://10.133.0.228:8000/ready\": dial tcp 10.133.0.228:8000: connect: connection refused" Oct 30 20:33:07.292659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.292262 5733 generic.go:356] "Generic (PLEG): container finished" podID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerID="ee3c574c192368417035048d601a4875afe8b025ebd33839c86dcd2c92b8c816" exitCode=0 Oct 30 20:33:07.292659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.292412 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" event={"ID":"fb50fef2-380e-4357-af32-6dcb3ec36f32","Type":"ContainerDied","Data":"ee3c574c192368417035048d601a4875afe8b025ebd33839c86dcd2c92b8c816"} Oct 30 20:33:07.395251 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.395220 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490369 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490442 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490554 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490600 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490660 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.494132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.490713 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\" (UniqueName: \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\") pod \"fb50fef2-380e-4357-af32-6dcb3ec36f32\" (UID: \"fb50fef2-380e-4357-af32-6dcb3ec36f32\") " Oct 30 20:33:07.509988 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.509786 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:07.510890 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.509905 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "kube-api-access-dh6ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:33:07.510890 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.510016 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:07.516083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.513147 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:07.516083 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.513246 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:33:07.535280 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.535244 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs" pod "fb50fef2-380e-4357-af32-6dcb3ec36f32" (UID: "fb50fef2-380e-4357-af32-6dcb3ec36f32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592533 5733 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-model-cache\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592572 5733 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb50fef2-380e-4357-af32-6dcb3ec36f32-tls-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592612 5733 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-tmp-dir\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592628 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6ql\" (UniqueName: \"kubernetes.io/projected/fb50fef2-380e-4357-af32-6dcb3ec36f32-kube-api-access-dh6ql\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592643 5733 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-dshm\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:07.592729 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:07.592658 5733 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb50fef2-380e-4357-af32-6dcb3ec36f32-home\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:33:08.299426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:08.299333 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" event={"ID":"fb50fef2-380e-4357-af32-6dcb3ec36f32","Type":"ContainerDied","Data":"a7df5975e86742f0bcdfeb65f736f1d13d227d0080aaa642dd3f8126581d641c"} Oct 30 20:33:08.299426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:08.299391 5733 scope.go:122] "RemoveContainer" containerID="ee3c574c192368417035048d601a4875afe8b025ebd33839c86dcd2c92b8c816" Oct 30 20:33:08.299426 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:08.299352 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2" Oct 30 20:33:08.318636 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:08.318609 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:33:08.323583 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:08.323557 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-381a9150-kserve-5f996bbf69-brks2"] Oct 30 20:33:10.089662 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:10.089204 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" path="/var/lib/kubelet/pods/fb50fef2-380e-4357-af32-6dcb3ec36f32/volumes" Oct 30 20:33:21.027173 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:21.026934 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99"] Oct 30 20:33:21.031047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:21.031021 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5rd99"] Oct 30 20:33:22.049520 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:22.049483 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c738da4-b62d-4a98-9dab-2054fde8ecc2" path="/var/lib/kubelet/pods/4c738da4-b62d-4a98-9dab-2054fde8ecc2/volumes" Oct 30 20:33:27.707747 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.707708 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-v94hc"] Oct 30 20:33:27.708349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708327 5733 cpu_manager.go:414] "containerMap: removing container" podUID="08d2ad8b-f5a0-4829-8003-600a14baaa6d" containerName="revoke-keys" Oct 30 20:33:27.708441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708351 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d2ad8b-f5a0-4829-8003-600a14baaa6d" containerName="revoke-keys" Oct 30 20:33:27.708441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708404 5733 cpu_manager.go:414] "containerMap: removing container" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerName="main" Oct 30 20:33:27.708441 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708414 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerName="main" Oct 30 20:33:27.708603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708506 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d2ad8b-f5a0-4829-8003-600a14baaa6d" containerName="revoke-keys" Oct 30 20:33:27.708603 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.708526 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb50fef2-380e-4357-af32-6dcb3ec36f32" containerName="main" Oct 30 20:33:27.713016 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.712994 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.721345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.720993 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-v94hc"] Oct 30 20:33:27.731668 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.731641 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-openshift-service-ca\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.731792 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.731707 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-metrics-certs\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.731850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.731823 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-maas-api-tls\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.731906 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.731890 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xb7x\" (UniqueName: \"kubernetes.io/projected/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-kube-api-access-4xb7x\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.790882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.790855 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 20:33:27.790882 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.790867 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 20:33:27.818028 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.817992 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 20:33:27.818193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.818051 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-47lpr\"" Oct 30 20:33:27.818193 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.818095 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 20:33:27.832528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.832501 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-openshift-service-ca\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.832686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.832545 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-metrics-certs\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.832686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.832573 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-maas-api-tls\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.832686 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.832602 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xb7x\" (UniqueName: \"kubernetes.io/projected/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-kube-api-access-4xb7x\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.833453 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.833429 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-openshift-service-ca\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.835234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.835212 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-maas-api-tls\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.835342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.835259 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-metrics-certs\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:27.844333 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:27.844307 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xb7x\" (UniqueName: \"kubernetes.io/projected/2b6cd03e-b987-48a2-a5ec-c6d2f7076843-kube-api-access-4xb7x\") pod \"maas-api-68445fb9db-v94hc\" (UID: \"2b6cd03e-b987-48a2-a5ec-c6d2f7076843\") " pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:28.028269 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:28.028180 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:28.312877 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:28.312846 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68445fb9db-v94hc"] Oct 30 20:33:28.314352 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:33:28.314324 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b6cd03e_b987_48a2_a5ec_c6d2f7076843.slice/crio-99fa3ed6d025aa953f17347838d28324197823aa9db355a1382727d722b659c7 WatchSource:0}: Error finding container 99fa3ed6d025aa953f17347838d28324197823aa9db355a1382727d722b659c7: Status 404 returned error can't find the container with id 99fa3ed6d025aa953f17347838d28324197823aa9db355a1382727d722b659c7 Oct 30 20:33:28.316393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:28.316368 5733 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:33:28.424759 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:28.424720 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" event={"ID":"2b6cd03e-b987-48a2-a5ec-c6d2f7076843","Type":"ContainerStarted","Data":"99fa3ed6d025aa953f17347838d28324197823aa9db355a1382727d722b659c7"} Oct 30 20:33:29.376558 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:33:29.376521 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:33:29.431226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:29.431195 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" event={"ID":"2b6cd03e-b987-48a2-a5ec-c6d2f7076843","Type":"ContainerStarted","Data":"45818b49396387a168625cb4a189db58ad33407085520dae240b0537b353bb4e"} Oct 30 20:33:29.431379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:29.431252 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:29.453119 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:29.453074 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" podStartSLOduration=2.046483293 podStartE2EDuration="2.453045043s" totalImagesPullingTime="406.56175ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:29.449973361 +0000 UTC m=+3769.517808932" watchObservedRunningTime="2026-10-30 20:33:29.453045043 +0000 UTC m=+3769.520880616" Oct 30 20:33:34.836128 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:34.836094 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:35.418628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:35.418590 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:35.442425 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:35.442396 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68445fb9db-v94hc" Oct 30 20:33:35.578788 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:35.578757 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:38.579073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:38.579035 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:38.764090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:38.764038 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:39.823850 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.823815 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:33:39.829025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.828993 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:39.832971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.832942 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:39.838002 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.837980 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-33631e52-serving-cert\"" Oct 30 20:33:39.845224 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.845193 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:33:39.866095 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.866047 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:33:39.869936 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.869914 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:39.879234 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.879208 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-33631e52\"" Oct 30 20:33:39.879416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.879395 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-33631e52-dockercfg-xgspv\"" Oct 30 20:33:39.884795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.884770 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:33:39.915961 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.915927 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:33:39.920621 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.920597 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:39.922616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.922587 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:33:39.961756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961729 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:39.961883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961768 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:39.961883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961786 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:39.961883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961858 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n8pj\" (UniqueName: \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:39.961997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961912 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:39.961997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.961935 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qct6\" (UniqueName: \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:39.999235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:39.999208 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:40.063192 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063160 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:40.063388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063207 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.063388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063339 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.063527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063382 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.063527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063420 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n8pj\" (UniqueName: \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:40.063527 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063474 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.063730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063541 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qct6\" (UniqueName: \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.063730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.063667 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbmfl\" (UniqueName: \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.063730 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:33:40.063714 5733 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-serving-cert: secret "maas-api-e2e-ait-33631e52-serving-cert" not found Oct 30 20:33:40.063908 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:33:40.063789 5733 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls podName:ce291548-ade1-437d-9868-4d8d50826fbe nodeName:}" failed. No retries permitted until 2026-10-30 20:33:40.563767921 +0000 UTC m=+3780.631603481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls") pod "maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" (UID: "ce291548-ade1-437d-9868-4d8d50826fbe") : secret "maas-api-e2e-ait-33631e52-serving-cert" not found Oct 30 20:33:40.064048 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.064021 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:40.064174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.064108 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.065740 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.065719 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.080893 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.080828 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n8pj\" (UniqueName: \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\") pod \"payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:40.082004 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.081985 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qct6\" (UniqueName: \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.164998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.164969 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbmfl\" (UniqueName: \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.165188 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.165089 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.165751 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.165728 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.183386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.183359 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:40.185627 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.185601 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbmfl\" (UniqueName: \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\") pod \"payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.237376 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.237258 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:40.285953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.285881 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:40.461873 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.461842 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:33:40.463467 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:33:40.463440 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99b1694_e282_4c46_9643_bc33a628c3d2.slice/crio-88b1635e071f06757f108f94336b2b0ea5baf3e6a7008afd770349d8938a12bf WatchSource:0}: Error finding container 88b1635e071f06757f108f94336b2b0ea5baf3e6a7008afd770349d8938a12bf: Status 404 returned error can't find the container with id 88b1635e071f06757f108f94336b2b0ea5baf3e6a7008afd770349d8938a12bf Oct 30 20:33:40.496542 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.496515 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" event={"ID":"d99b1694-e282-4c46-9643-bc33a628c3d2","Type":"ContainerStarted","Data":"88b1635e071f06757f108f94336b2b0ea5baf3e6a7008afd770349d8938a12bf"} Oct 30 20:33:40.534274 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.534250 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:33:40.535265 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:33:40.535228 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18942690_cf64_4083_abab_0d47ee3197b9.slice/crio-98567e1a5699edfa1305ac09bf53c7e5b98bbcf7e07040ecc00f6e7782827e94 WatchSource:0}: Error finding container 98567e1a5699edfa1305ac09bf53c7e5b98bbcf7e07040ecc00f6e7782827e94: Status 404 returned error can't find the container with id 98567e1a5699edfa1305ac09bf53c7e5b98bbcf7e07040ecc00f6e7782827e94 Oct 30 20:33:40.569152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.569122 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.571712 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.571687 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") pod \"maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.749345 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.749263 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:40.907776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:40.907729 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:41.031579 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.031552 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:33:41.033097 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:33:41.033050 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce291548_ade1_437d_9868_4d8d50826fbe.slice/crio-46e97c84a605f7643b824797579b5760fbd92435a1f9f9b7e9fe580138422b14 WatchSource:0}: Error finding container 46e97c84a605f7643b824797579b5760fbd92435a1f9f9b7e9fe580138422b14: Status 404 returned error can't find the container with id 46e97c84a605f7643b824797579b5760fbd92435a1f9f9b7e9fe580138422b14 Oct 30 20:33:41.505643 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.505610 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" event={"ID":"18942690-cf64-4083-abab-0d47ee3197b9","Type":"ContainerStarted","Data":"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486"} Oct 30 20:33:41.505775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.505714 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" event={"ID":"18942690-cf64-4083-abab-0d47ee3197b9","Type":"ContainerStarted","Data":"98567e1a5699edfa1305ac09bf53c7e5b98bbcf7e07040ecc00f6e7782827e94"} Oct 30 20:33:41.505775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.505743 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:41.507374 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.507350 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" event={"ID":"d99b1694-e282-4c46-9643-bc33a628c3d2","Type":"ContainerStarted","Data":"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526"} Oct 30 20:33:41.507487 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.507409 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:41.508671 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.508649 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" event={"ID":"ce291548-ade1-437d-9868-4d8d50826fbe","Type":"ContainerStarted","Data":"46e97c84a605f7643b824797579b5760fbd92435a1f9f9b7e9fe580138422b14"} Oct 30 20:33:41.527490 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.527440 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" podStartSLOduration=2.527424103 podStartE2EDuration="2.527424103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:41.52366498 +0000 UTC m=+3781.591500548" watchObservedRunningTime="2026-10-30 20:33:41.527424103 +0000 UTC m=+3781.595259671" Oct 30 20:33:41.543761 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.543687 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" podStartSLOduration=2.5436701360000002 podStartE2EDuration="2.543670136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:41.541233824 +0000 UTC m=+3781.609069393" watchObservedRunningTime="2026-10-30 20:33:41.543670136 +0000 UTC m=+3781.611505708" Oct 30 20:33:41.904804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:41.904770 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:42.518125 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:42.518023 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" event={"ID":"ce291548-ade1-437d-9868-4d8d50826fbe","Type":"ContainerStarted","Data":"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d"} Oct 30 20:33:42.539342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:42.539291 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" podStartSLOduration=3.129744782 podStartE2EDuration="3.539272246s" totalImagesPullingTime="409.527464ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:33:42.537604735 +0000 UTC m=+3782.605440302" watchObservedRunningTime="2026-10-30 20:33:42.539272246 +0000 UTC m=+3782.607107816" Oct 30 20:33:42.827773 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:42.827695 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:43.524312 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:43.524280 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:43.593760 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:43.593719 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:44.410989 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:44.410936 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:45.515651 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:45.515597 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:46.277397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:46.277311 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:47.092235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:47.092195 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:47.987587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:47.987555 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:48.884012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:48.883970 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:49.535329 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:49.535298 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:33:49.695657 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:49.695625 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:50.486502 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:50.486462 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:51.389402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:51.389282 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:52.389776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:52.389740 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:52.520997 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:52.520970 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:33:52.521220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:52.521206 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:33:53.193595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:53.193560 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:54.080194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:54.080119 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:54.223950 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:54.223914 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:54.992383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:54.992345 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:55.244395 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:55.244294 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:55.925077 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:55.925024 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:56.819771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:56.819728 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:57.609595 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:57.609563 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:58.401846 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:58.401812 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:59.359588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:33:59.359557 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:33:59.376705 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:33:59.376671 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:34:00.107475 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:00.107440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:00.910675 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:00.910635 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:01.708040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:01.707995 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:02.592610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:02.592574 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:03.422555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:03.422517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:04.306661 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:04.306627 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:04.600488 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:04.600448 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:34:04.619040 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:04.619009 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:34:04.717048 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:04.716993 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:04.873383 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:04.873281 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:05.179139 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:05.179043 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:05.788506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:05.788473 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:34:05.789091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:05.788692 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" podUID="ce291548-ade1-437d-9868-4d8d50826fbe" containerName="maas-api" containerID="cri-o://0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d" gracePeriod=30 Oct 30 20:34:05.896898 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:05.896858 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:06.061164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.061134 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:34:06.100026 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.099983 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:06.120748 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.120717 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\") pod \"ce291548-ade1-437d-9868-4d8d50826fbe\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " Oct 30 20:34:06.120933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.120762 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\" (UniqueName: \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\") pod \"ce291548-ade1-437d-9868-4d8d50826fbe\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " Oct 30 20:34:06.120933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.120804 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") pod \"ce291548-ade1-437d-9868-4d8d50826fbe\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " Oct 30 20:34:06.120933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.120834 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\") pod \"ce291548-ade1-437d-9868-4d8d50826fbe\" (UID: \"ce291548-ade1-437d-9868-4d8d50826fbe\") " Oct 30 20:34:06.121175 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.121146 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca" pod "ce291548-ade1-437d-9868-4d8d50826fbe" (UID: "ce291548-ade1-437d-9868-4d8d50826fbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:06.121815 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.121787 5733 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce291548-ade1-437d-9868-4d8d50826fbe-openshift-service-ca\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:06.123152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.123125 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls" pod "ce291548-ade1-437d-9868-4d8d50826fbe" (UID: "ce291548-ade1-437d-9868-4d8d50826fbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:06.123236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.123148 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6" pod "ce291548-ade1-437d-9868-4d8d50826fbe" (UID: "ce291548-ade1-437d-9868-4d8d50826fbe"). InnerVolumeSpecName "kube-api-access-6qct6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:06.123236 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.123162 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs" pod "ce291548-ade1-437d-9868-4d8d50826fbe" (UID: "ce291548-ade1-437d-9868-4d8d50826fbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:34:06.222998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.222961 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qct6\" (UniqueName: \"kubernetes.io/projected/ce291548-ade1-437d-9868-4d8d50826fbe-kube-api-access-6qct6\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:06.222998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.222993 5733 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-maas-api-tls\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:06.222998 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.223004 5733 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce291548-ade1-437d-9868-4d8d50826fbe-metrics-certs\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:06.267912 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.267868 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:06.670096 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.670044 5733 generic.go:356] "Generic (PLEG): container finished" podID="ce291548-ade1-437d-9868-4d8d50826fbe" containerID="0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d" exitCode=0 Oct 30 20:34:06.670281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.670115 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" Oct 30 20:34:06.670281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.670122 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" event={"ID":"ce291548-ade1-437d-9868-4d8d50826fbe","Type":"ContainerDied","Data":"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d"} Oct 30 20:34:06.670281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.670248 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg" event={"ID":"ce291548-ade1-437d-9868-4d8d50826fbe","Type":"ContainerDied","Data":"46e97c84a605f7643b824797579b5760fbd92435a1f9f9b7e9fe580138422b14"} Oct 30 20:34:06.670281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.670270 5733 scope.go:122] "RemoveContainer" containerID="0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d" Oct 30 20:34:06.680485 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.680468 5733 scope.go:122] "RemoveContainer" containerID="0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d" Oct 30 20:34:06.680773 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:34:06.680754 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d\": container with ID starting with 0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d not found: ID does not exist" containerID="0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d" Oct 30 20:34:06.680835 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.680782 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d"} err="failed to get container status \"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d\": rpc error: code = NotFound desc = could not find container \"0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d\": container with ID starting with 0f22583ca52674d1a2cdc6f06346f9d6715285a58eca86e43aef06449979880d not found: ID does not exist" Oct 30 20:34:06.699265 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.699238 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:34:06.703934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:06.703912 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33631e52-d8c94dd5c-x2xjg"] Oct 30 20:34:08.049643 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:08.049611 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce291548-ade1-437d-9868-4d8d50826fbe" path="/var/lib/kubelet/pods/ce291548-ade1-437d-9868-4d8d50826fbe/volumes" Oct 30 20:34:09.601094 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.601044 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" podUID="18942690-cf64-4083-abab-0d47ee3197b9" containerName="payload-processing" containerID="cri-o://f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486" gracePeriod=25 Oct 30 20:34:09.620044 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.620000 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" podUID="d99b1694-e282-4c46-9643-bc33a628c3d2" containerName="payload-pre-processing" containerID="cri-o://fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526" gracePeriod=25 Oct 30 20:34:09.877042 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.877015 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:34:09.907795 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.907768 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:34:09.964632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.964604 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\") pod \"18942690-cf64-4083-abab-0d47ee3197b9\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " Oct 30 20:34:09.964813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.964653 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\") pod \"d99b1694-e282-4c46-9643-bc33a628c3d2\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " Oct 30 20:34:09.964813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.964691 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\" (UniqueName: \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\") pod \"d99b1694-e282-4c46-9643-bc33a628c3d2\" (UID: \"d99b1694-e282-4c46-9643-bc33a628c3d2\") " Oct 30 20:34:09.964813 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.964741 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\" (UniqueName: \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\") pod \"18942690-cf64-4083-abab-0d47ee3197b9\" (UID: \"18942690-cf64-4083-abab-0d47ee3197b9\") " Oct 30 20:34:09.965005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.964981 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume" pod "18942690-cf64-4083-abab-0d47ee3197b9" (UID: "18942690-cf64-4083-abab-0d47ee3197b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:09.965143 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.965122 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume" pod "d99b1694-e282-4c46-9643-bc33a628c3d2" (UID: "d99b1694-e282-4c46-9643-bc33a628c3d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:34:09.965220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.965170 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18942690-cf64-4083-abab-0d47ee3197b9-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:09.966905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.966874 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl" pod "18942690-cf64-4083-abab-0d47ee3197b9" (UID: "18942690-cf64-4083-abab-0d47ee3197b9"). InnerVolumeSpecName "kube-api-access-nbmfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:09.967000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:09.966963 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj" pod "d99b1694-e282-4c46-9643-bc33a628c3d2" (UID: "d99b1694-e282-4c46-9643-bc33a628c3d2"). InnerVolumeSpecName "kube-api-access-2n8pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:34:10.066491 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.066464 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d99b1694-e282-4c46-9643-bc33a628c3d2-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:10.066491 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.066491 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n8pj\" (UniqueName: \"kubernetes.io/projected/d99b1694-e282-4c46-9643-bc33a628c3d2-kube-api-access-2n8pj\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:10.066660 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.066501 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbmfl\" (UniqueName: \"kubernetes.io/projected/18942690-cf64-4083-abab-0d47ee3197b9-kube-api-access-nbmfl\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:34:10.697447 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.697412 5733 generic.go:356] "Generic (PLEG): container finished" podID="18942690-cf64-4083-abab-0d47ee3197b9" containerID="f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486" exitCode=0 Oct 30 20:34:10.697872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.697470 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" Oct 30 20:34:10.697872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.697494 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" event={"ID":"18942690-cf64-4083-abab-0d47ee3197b9","Type":"ContainerDied","Data":"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486"} Oct 30 20:34:10.697872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.697585 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg" event={"ID":"18942690-cf64-4083-abab-0d47ee3197b9","Type":"ContainerDied","Data":"98567e1a5699edfa1305ac09bf53c7e5b98bbcf7e07040ecc00f6e7782827e94"} Oct 30 20:34:10.697872 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.697620 5733 scope.go:122] "RemoveContainer" containerID="f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486" Oct 30 20:34:10.699642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.699449 5733 generic.go:356] "Generic (PLEG): container finished" podID="d99b1694-e282-4c46-9643-bc33a628c3d2" containerID="fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526" exitCode=0 Oct 30 20:34:10.699642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.699549 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" event={"ID":"d99b1694-e282-4c46-9643-bc33a628c3d2","Type":"ContainerDied","Data":"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526"} Oct 30 20:34:10.699642 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.699585 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" event={"ID":"d99b1694-e282-4c46-9643-bc33a628c3d2","Type":"ContainerDied","Data":"88b1635e071f06757f108f94336b2b0ea5baf3e6a7008afd770349d8938a12bf"} Oct 30 20:34:10.699839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.699670 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w" Oct 30 20:34:10.707845 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.707816 5733 scope.go:122] "RemoveContainer" containerID="f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486" Oct 30 20:34:10.708216 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:34:10.708194 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486\": container with ID starting with f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486 not found: ID does not exist" containerID="f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486" Oct 30 20:34:10.708317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.708226 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486"} err="failed to get container status \"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486\": rpc error: code = NotFound desc = could not find container \"f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486\": container with ID starting with f1b9ccf658983de39654b53537e2d6b1961748a8cfc9e15ecd8f3c90e3522486 not found: ID does not exist" Oct 30 20:34:10.708317 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.708248 5733 scope.go:122] "RemoveContainer" containerID="fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526" Oct 30 20:34:10.717102 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.717075 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:34:10.719226 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.719205 5733 scope.go:122] "RemoveContainer" containerID="fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526" Oct 30 20:34:10.719500 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:34:10.719466 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526\": container with ID starting with fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526 not found: ID does not exist" containerID="fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526" Oct 30 20:34:10.719580 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.719508 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526"} err="failed to get container status \"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526\": rpc error: code = NotFound desc = could not find container \"fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526\": container with ID starting with fe0850ea17bc952a6bbd7790fa64b9b343f77f289d6c456923bb4ce1f627c526 not found: ID does not exist" Oct 30 20:34:10.721511 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.721489 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-33631e52-6f7cc98cbb-ld2lg"] Oct 30 20:34:10.733959 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.733931 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:34:10.739896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:10.739872 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-33631e52-6cff7957d7-8gx2w"] Oct 30 20:34:11.096610 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:11.096582 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:12.048470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:12.048439 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18942690-cf64-4083-abab-0d47ee3197b9" path="/var/lib/kubelet/pods/18942690-cf64-4083-abab-0d47ee3197b9/volumes" Oct 30 20:34:12.048947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:12.048817 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99b1694-e282-4c46-9643-bc33a628c3d2" path="/var/lib/kubelet/pods/d99b1694-e282-4c46-9643-bc33a628c3d2/volumes" Oct 30 20:34:12.464782 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:12.464750 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:12.918973 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:12.918916 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:13.116233 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:13.116178 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:16.901011 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:16.900972 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:17.149473 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:17.149440 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:18.146953 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.146919 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:34:18.147556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147529 5733 cpu_manager.go:414] "containerMap: removing container" podUID="d99b1694-e282-4c46-9643-bc33a628c3d2" containerName="payload-pre-processing" Oct 30 20:34:18.147556 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147554 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99b1694-e282-4c46-9643-bc33a628c3d2" containerName="payload-pre-processing" Oct 30 20:34:18.147737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147591 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ce291548-ade1-437d-9868-4d8d50826fbe" containerName="maas-api" Oct 30 20:34:18.147737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147598 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce291548-ade1-437d-9868-4d8d50826fbe" containerName="maas-api" Oct 30 20:34:18.147737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147620 5733 cpu_manager.go:414] "containerMap: removing container" podUID="18942690-cf64-4083-abab-0d47ee3197b9" containerName="payload-processing" Oct 30 20:34:18.147737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147629 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18942690-cf64-4083-abab-0d47ee3197b9" containerName="payload-processing" Oct 30 20:34:18.147737 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147735 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99b1694-e282-4c46-9643-bc33a628c3d2" containerName="payload-pre-processing" Oct 30 20:34:18.147954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147753 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce291548-ade1-437d-9868-4d8d50826fbe" containerName="maas-api" Oct 30 20:34:18.147954 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.147764 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="18942690-cf64-4083-abab-0d47ee3197b9" containerName="payload-processing" Oct 30 20:34:18.148398 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.148378 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:18.151309 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.151292 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:34:18.151430 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.151415 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.154718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.154698 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.160827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.160804 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-48059e\"" Oct 30 20:34:18.163331 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.163307 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:34:18.165007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.164983 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-48059e-dockercfg-w5gd5\"" Oct 30 20:34:18.166933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.166916 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:34:18.249781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.249749 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbd9z\" (UniqueName: \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.249962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.249816 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.249962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.249926 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.249962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.249948 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmm7t\" (UniqueName: \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.329217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.329186 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:18.351342 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.351308 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.351555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.351349 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmm7t\" (UniqueName: \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.351555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.351410 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbd9z\" (UniqueName: \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.351555 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.351454 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.352167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.352140 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.352288 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.352271 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.365870 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.365848 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbd9z\" (UniqueName: \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\") pod \"payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.366090 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.366071 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmm7t\" (UniqueName: \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\") pod \"payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.468476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.468404 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:18.480649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.480626 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:18.483999 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.483974 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:18.759080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.759010 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:34:18.760848 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:34:18.760815 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d837f4e_68b7_4654_ab4e_8d16c253a358.slice/crio-fd65e10d990670e87e2225276d1dafa1daff29180b811bee1497d36478d6a42b WatchSource:0}: Error finding container fd65e10d990670e87e2225276d1dafa1daff29180b811bee1497d36478d6a42b: Status 404 returned error can't find the container with id fd65e10d990670e87e2225276d1dafa1daff29180b811bee1497d36478d6a42b Oct 30 20:34:18.779483 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:18.779395 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:34:18.795386 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:34:18.795359 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded988cf9_4a6a_46ed_b0f7_dee6c9c40a6b.slice/crio-139cd2f1f3d1426d9ee3c354530d510207c7f24b2ae1b53347ae0faa6a42086e WatchSource:0}: Error finding container 139cd2f1f3d1426d9ee3c354530d510207c7f24b2ae1b53347ae0faa6a42086e: Status 404 returned error can't find the container with id 139cd2f1f3d1426d9ee3c354530d510207c7f24b2ae1b53347ae0faa6a42086e Oct 30 20:34:19.197310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.197272 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:19.757889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.757851 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" event={"ID":"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b","Type":"ContainerStarted","Data":"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff"} Oct 30 20:34:19.758117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.757927 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" event={"ID":"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b","Type":"ContainerStarted","Data":"139cd2f1f3d1426d9ee3c354530d510207c7f24b2ae1b53347ae0faa6a42086e"} Oct 30 20:34:19.758117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.758091 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:19.760005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.759961 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" event={"ID":"5d837f4e-68b7-4654-ab4e-8d16c253a358","Type":"ContainerStarted","Data":"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02"} Oct 30 20:34:19.760220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.760020 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" event={"ID":"5d837f4e-68b7-4654-ab4e-8d16c253a358","Type":"ContainerStarted","Data":"fd65e10d990670e87e2225276d1dafa1daff29180b811bee1497d36478d6a42b"} Oct 30 20:34:19.760220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.760053 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:19.777229 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.777177 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" podStartSLOduration=1.777161153 podStartE2EDuration="1.777161153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:19.775503606 +0000 UTC m=+3819.843339175" watchObservedRunningTime="2026-10-30 20:34:19.777161153 +0000 UTC m=+3819.844996722" Oct 30 20:34:19.794091 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:19.794030 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" podStartSLOduration=1.794022191 podStartE2EDuration="1.794022191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:34:19.793136897 +0000 UTC m=+3819.860972466" watchObservedRunningTime="2026-10-30 20:34:19.794022191 +0000 UTC m=+3819.861857759" Oct 30 20:34:20.105763 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:20.105711 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:21.088552 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:21.088518 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:22.102211 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:22.102156 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:23.009733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:23.009699 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:23.374538 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:23.374499 5733 scope.go:122] "RemoveContainer" containerID="c4535ca01be9b7bb0f6dc75339ef29b8fa78ba3f29590a27f1ba2449f2c0e967" Oct 30 20:34:23.944150 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:23.944110 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:24.808974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:24.808928 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:25.710325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:25.710292 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:26.831071 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:26.831027 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:27.709473 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:27.709446 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:28.535194 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:28.535161 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:29.366421 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:29.366390 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:29.376457 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:34:29.376429 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:34:30.085572 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:30.085534 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:30.770221 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:30.770190 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:34:30.770407 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:30.770268 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:34:30.897862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:30.897832 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:31.803431 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:31.803385 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:31.983225 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:31.983194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:32.729858 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:32.729820 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:33.617346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:33.617292 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:34.710528 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:34.710500 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:35.607384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:35.607350 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:36.495626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:36.495537 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:37.294325 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:37.294292 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:38.125178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:38.125148 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:38.877359 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:38.877323 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:39.688007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:39.687975 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:40.489566 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:40.489532 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:41.301725 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:41.301676 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:42.143217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:42.143181 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:43.010944 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:43.010904 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:43.794408 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:43.794375 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:44.695711 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:44.695677 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:45.498201 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:45.498164 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:46.312801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:46.312765 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:47.272476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:47.272426 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:48.062801 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:48.062766 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:48.877765 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:48.877710 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:49.753035 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:49.752999 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:50.607318 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:50.607270 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:51.471883 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:51.471841 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:52.335707 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:52.335672 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:53.112493 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:53.112408 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:53.902894 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:53.902860 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:54.116617 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:54.116588 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:54.694281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:54.694238 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:55.298730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:55.298692 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:55.484567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:55.484511 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:56.386241 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:56.386189 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:57.190986 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:57.190939 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:57.982948 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:57.982914 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:58.804785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:58.804704 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:34:59.377424 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:34:59.377383 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:34:59.628754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:34:59.628676 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:00.396664 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:00.396624 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:01.197508 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:01.197469 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:02.019543 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:02.019512 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:02.798198 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:02.798132 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:03.601756 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:03.601723 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:04.543824 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:04.543766 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:05.397569 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:05.397497 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:06.173307 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:06.173252 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:07.011694 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:07.011659 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:07.808535 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:07.808505 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:08.587379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:08.587344 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:09.486098 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:09.486046 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:10.300670 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:10.300630 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:11.130577 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:11.130546 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:11.989223 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:11.989187 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:12.049311 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:12.049280 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz"] Oct 30 20:35:12.053300 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:12.053274 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-vbxnz"] Oct 30 20:35:12.881669 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:12.881632 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:13.784025 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:13.783987 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:14.048981 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:14.048811 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b23f730-181a-4c99-94c2-8ce7d1253f0d" path="/var/lib/kubelet/pods/5b23f730-181a-4c99-94c2-8ce7d1253f0d/volumes" Oct 30 20:35:14.722616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:14.722578 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:15.591781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:15.591711 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:16.506937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:16.506900 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:17.321692 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:17.321654 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:18.087859 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:18.087830 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:18.883964 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:18.883929 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:19.786741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:19.786711 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:20.593217 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:20.593173 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:21.385980 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:21.385899 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:22.285615 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:22.285580 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:23.095115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:23.095075 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:23.893817 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:23.893782 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:24.796680 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:24.796645 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:25.581826 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:25.581793 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:26.486115 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:26.486050 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:27.288356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:27.288305 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:28.083379 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:28.083338 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:28.915779 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:28.915749 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:29.377628 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:35:29.377590 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:35:29.857326 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:29.857289 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:30.580253 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:30.580217 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:31.480235 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:31.480201 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:32.282811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:32.282777 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:33.103163 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:33.103135 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:33.908450 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:33.908416 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:34.807831 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:34.807791 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:35.589830 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:35.589800 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:36.404051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:36.404018 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:37.191349 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:37.191313 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:38.012548 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:38.012515 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:38.916263 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:38.916228 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:39.719393 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:39.719355 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:40.578024 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:40.577991 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:41.068962 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:41.068934 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:35:41.081971 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:41.081948 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-ng7cn_d8efd1a0-6c9c-4f9a-bbb8-2a280f2b94d5/csi-driver/0.log" Oct 30 20:35:41.380533 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:41.380499 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:42.287770 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:42.287736 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:43.075839 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:43.075785 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:43.882020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:43.881986 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:44.684976 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:44.684934 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:45.492867 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:45.492833 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:46.283178 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:46.283142 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:47.107228 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:47.107194 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:47.890754 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:47.890707 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:48.933117 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:48.933079 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:49.854623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:49.854585 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:50.809784 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:50.809752 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:51.592733 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:51.592679 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:52.233582 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:52.233534 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:52.377160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:52.377111 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:52.519051 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:52.518948 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:53.379641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:53.379603 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:54.176106 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:54.176075 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:54.316205 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:54.316168 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:55.157231 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:55.157007 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:55.350940 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:55.350898 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:55.982416 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:55.982377 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:56.799746 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:56.799703 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:57.631434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:57.631399 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:58.824852 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:58.824763 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:59.263512 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:35:59.263474 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:35:59.375621 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:35:59.375586 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:36:00.008559 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:00.008517 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:01.000785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:01.000738 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:01.817544 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:01.817507 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:02.415629 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:02.411694 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:36:02.420974 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:02.420937 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:36:02.549587 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:02.549546 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:02.683122 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:02.682971 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:02.830391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:02.830356 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:07.412219 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.412182 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" podUID="5d837f4e-68b7-4654-ab4e-8d16c253a358" containerName="payload-processing" containerID="cri-o://8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02" gracePeriod=25 Oct 30 20:36:07.421623 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.421589 5733 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" podUID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" containerName="payload-pre-processing" containerID="cri-o://fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff" gracePeriod=25 Oct 30 20:36:07.670541 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.670480 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:36:07.709467 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.709438 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:36:07.730771 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.730734 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\") pod \"5d837f4e-68b7-4654-ab4e-8d16c253a358\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " Oct 30 20:36:07.730968 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.730935 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\" (UniqueName: \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\") pod \"5d837f4e-68b7-4654-ab4e-8d16c253a358\" (UID: \"5d837f4e-68b7-4654-ab4e-8d16c253a358\") " Oct 30 20:36:07.731683 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.731657 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume" pod "5d837f4e-68b7-4654-ab4e-8d16c253a358" (UID: "5d837f4e-68b7-4654-ab4e-8d16c253a358"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:07.733434 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.733410 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t" pod "5d837f4e-68b7-4654-ab4e-8d16c253a358" (UID: "5d837f4e-68b7-4654-ab4e-8d16c253a358"). InnerVolumeSpecName "kube-api-access-hmm7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:07.832433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.832402 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\") pod \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " Oct 30 20:36:07.832649 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.832471 5733 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\" (UniqueName: \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\") pod \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\" (UID: \"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b\") " Oct 30 20:36:07.832785 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.832763 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume" pod "ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" (UID: "ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:36:07.832868 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.832849 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmm7t\" (UniqueName: \"kubernetes.io/projected/5d837f4e-68b7-4654-ab4e-8d16c253a358-kube-api-access-hmm7t\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:07.832920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.832875 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d837f4e-68b7-4654-ab4e-8d16c253a358-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:07.834509 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.834489 5733 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z" pod "ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" (UID: "ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b"). InnerVolumeSpecName "kube-api-access-kbd9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:36:07.933554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.933476 5733 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-plugins-config-volume\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:07.933554 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:07.933505 5733 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbd9z\" (UniqueName: \"kubernetes.io/projected/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b-kube-api-access-kbd9z\") on node \"ip-10-0-139-103.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:36:08.373390 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.373344 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.410933 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.410893 5733 generic.go:356] "Generic (PLEG): container finished" podID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" containerID="fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff" exitCode=0 Oct 30 20:36:08.411114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.410945 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" Oct 30 20:36:08.411114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.410977 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" event={"ID":"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b","Type":"ContainerDied","Data":"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff"} Oct 30 20:36:08.411114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.411028 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg" event={"ID":"ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b","Type":"ContainerDied","Data":"139cd2f1f3d1426d9ee3c354530d510207c7f24b2ae1b53347ae0faa6a42086e"} Oct 30 20:36:08.411114 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.411050 5733 scope.go:122] "RemoveContainer" containerID="fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff" Oct 30 20:36:08.412355 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.412332 5733 generic.go:356] "Generic (PLEG): container finished" podID="5d837f4e-68b7-4654-ab4e-8d16c253a358" containerID="8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02" exitCode=0 Oct 30 20:36:08.412753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.412375 5733 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" Oct 30 20:36:08.412753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.412413 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" event={"ID":"5d837f4e-68b7-4654-ab4e-8d16c253a358","Type":"ContainerDied","Data":"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02"} Oct 30 20:36:08.412753 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.412452 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt" event={"ID":"5d837f4e-68b7-4654-ab4e-8d16c253a358","Type":"ContainerDied","Data":"fd65e10d990670e87e2225276d1dafa1daff29180b811bee1497d36478d6a42b"} Oct 30 20:36:08.421167 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.420958 5733 scope.go:122] "RemoveContainer" containerID="fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff" Oct 30 20:36:08.421324 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:36:08.421297 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff\": container with ID starting with fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff not found: ID does not exist" containerID="fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff" Oct 30 20:36:08.421396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.421335 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff"} err="failed to get container status \"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff\": rpc error: code = NotFound desc = could not find container \"fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff\": container with ID starting with fe4d96f8b1bd2deb04a9cdbd9471aa5ba708f4fc931b25cf22004b238707c8ff not found: ID does not exist" Oct 30 20:36:08.421396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.421367 5733 scope.go:122] "RemoveContainer" containerID="8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02" Oct 30 20:36:08.435914 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.435878 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:36:08.437565 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.437546 5733 scope.go:122] "RemoveContainer" containerID="8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02" Oct 30 20:36:08.437843 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:36:08.437823 5733 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02\": container with ID starting with 8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02 not found: ID does not exist" containerID="8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02" Oct 30 20:36:08.437896 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.437851 5733 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02"} err="failed to get container status \"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02\": rpc error: code = NotFound desc = could not find container \"8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02\": container with ID starting with 8645a81ce95cf2b7d7a309914a73390b93f787705cf317a11fe57f1655b2fe02 not found: ID does not exist" Oct 30 20:36:08.439174 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.439153 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-48059e-69959bb8c-9qllg"] Oct 30 20:36:08.449420 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.449398 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:36:08.458080 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.458030 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-48059e-578d8bbc4-7dgvt"] Oct 30 20:36:08.557745 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.557709 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:08.694840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:08.694755 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:09.203905 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:09.203873 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:10.049982 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:10.049938 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d837f4e-68b7-4654-ab4e-8d16c253a358" path="/var/lib/kubelet/pods/5d837f4e-68b7-4654-ab4e-8d16c253a358/volumes" Oct 30 20:36:10.050597 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:10.050572 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" path="/var/lib/kubelet/pods/ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b/volumes" Oct 30 20:36:15.444717 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:15.444681 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.581730 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:15.581655 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:15.874476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:15.874443 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:23.425452 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:23.425412 5733 scope.go:122] "RemoveContainer" containerID="e6d7d2171a16634ed3668640d69dd100044f596b32ee41823cd05dc0161fc007" Oct 30 20:36:29.377002 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:36:29.376961 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:36:34.032101 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:34.032047 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf"] Oct 30 20:36:34.041372 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:34.041333 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11a28650-vrmqf"] Oct 30 20:36:34.048774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:34.048743 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf72d073-8db1-48bf-82db-73a8fea145a0" path="/var/lib/kubelet/pods/cf72d073-8db1-48bf-82db-73a8fea145a0/volumes" Oct 30 20:36:51.839047 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:51.838931 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:54.234479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:54.234432 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:55.411885 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:36:55.411843 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:36:59.376098 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:36:59.376036 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:37:05.882320 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:05.882277 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log" Oct 30 20:37:15.140775 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:15.140717 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:15.289005 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:15.288976 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:19.184920 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:19.184889 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68445fb9db-v94hc_2b6cd03e-b987-48a2-a5ec-c6d2f7076843/maas-api/0.log" Oct 30 20:37:19.557497 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:19.557414 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-krf2t_08d2ad8b-f5a0-4829-8003-600a14baaa6d/revoke-keys/0.log" Oct 30 20:37:19.742445 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:19.742417 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-4wnhk_33ffe5fa-a599-46c0-a5b3-449df8067a1d/revoke-keys/0.log" Oct 30 20:37:22.027384 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:22.027356 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log" Oct 30 20:37:22.490823 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:22.490780 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/1.log" Oct 30 20:37:22.868836 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:22.868806 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-pcl5w_76e6166d-79fd-4d5c-adde-24ab3b2b0400/manager/0.log" Oct 30 20:37:23.732433 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:23.732388 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/0.log" Oct 30 20:37:24.696514 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:24.696479 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x4v2z_ff194676-b322-42b3-a970-7db28cc2ac3c/kube-auth-proxy/0.log" Oct 30 20:37:25.464402 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:25.464371 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/1.log" Oct 30 20:37:26.203626 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:26.203597 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ldpp9_393595d7-bbd7-4db8-b5a5-5db2cc93b72b/istio-proxy/0.log" Oct 30 20:37:26.373776 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:26.373725 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/1.log" Oct 30 20:37:27.304957 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:27.304927 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:37:28.131428 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:28.131402 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/0.log" Oct 30 20:37:28.692459 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:28.692430 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/0.log" Oct 30 20:37:29.376972 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:37:29.376931 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:37:29.623854 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:29.623825 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q9sb8_952eb089-1681-4eac-94d4-9f7465666913/main/0.log" Oct 30 20:37:29.814356 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:29.814266 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-nrlrs_cd58d3a0-2332-494d-85f0-e48f263ac29c/main/0.log" Oct 30 20:37:29.997082 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:29.997028 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tn875_294d869b-a6be-48ff-95f3-8123552c5d4f/main/0.log" Oct 30 20:37:30.541568 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:30.541528 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq_0c7a23be-1c57-4b4f-a494-0ff740d58fca/main/0.log" Oct 30 20:37:31.619972 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:31.619931 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log" Oct 30 20:37:33.916934 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:33.916896 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:34.068152 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:34.068121 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:42.024073 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:42.024030 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk"] Oct 30 20:37:42.027452 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:42.027420 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-4wnhk"] Oct 30 20:37:42.049409 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:42.049379 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ffe5fa-a599-46c0-a5b3-449df8067a1d" path="/var/lib/kubelet/pods/33ffe5fa-a599-46c0-a5b3-449df8067a1d/volumes" Oct 30 20:37:43.180840 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:43.180805 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68445fb9db-v94hc_2b6cd03e-b987-48a2-a5ec-c6d2f7076843/maas-api/0.log" Oct 30 20:37:43.560240 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:43.560156 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-krf2t_08d2ad8b-f5a0-4829-8003-600a14baaa6d/revoke-keys/0.log" Oct 30 20:37:45.425749 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:45.425713 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log" Oct 30 20:37:45.838131 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:45.838036 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/1.log" Oct 30 20:37:46.233012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:46.232971 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-pcl5w_76e6166d-79fd-4d5c-adde-24ab3b2b0400/manager/0.log" Oct 30 20:37:47.660937 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:47.660904 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x4v2z_ff194676-b322-42b3-a970-7db28cc2ac3c/kube-auth-proxy/0.log" Oct 30 20:37:48.412993 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:48.412964 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/1.log" Oct 30 20:37:49.169364 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:49.169325 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ldpp9_393595d7-bbd7-4db8-b5a5-5db2cc93b72b/istio-proxy/0.log" Oct 30 20:37:49.355536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:49.355506 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/1.log" Oct 30 20:37:50.262271 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:50.262222 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:37:51.179397 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:51.179347 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-q9sb8_952eb089-1681-4eac-94d4-9f7465666913/main/0.log" Oct 30 20:37:51.362628 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:51.362598 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-nrlrs_cd58d3a0-2332-494d-85f0-e48f263ac29c/main/0.log" Oct 30 20:37:51.536435 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:51.536355 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-tn875_294d869b-a6be-48ff-95f3-8123552c5d4f/main/0.log" Oct 30 20:37:52.081007 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:52.080976 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-zhtnq_0c7a23be-1c57-4b4f-a494-0ff740d58fca/main/0.log" Oct 30 20:37:54.446659 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:54.446626 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:55.479327 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:55.479287 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:37:59.024658 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:59.024618 5733 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t"] Oct 30 20:37:59.027827 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:37:59.027803 5733 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-krf2t"] Oct 30 20:37:59.377445 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:37:59.377409 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:38:00.050021 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:00.049974 5733 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d2ad8b-f5a0-4829-8003-600a14baaa6d" path="/var/lib/kubelet/pods/08d2ad8b-f5a0-4829-8003-600a14baaa6d/volumes" Oct 30 20:38:23.467138 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:23.467031 5733 scope.go:122] "RemoveContainer" containerID="0d25fd3c1c87d45d6dc480348a34348705a0eae1e4b77496ca3c1a03cdb370f2" Oct 30 20:38:23.476246 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:23.476227 5733 scope.go:122] "RemoveContainer" containerID="74495bfa43eea8d96c653609e3acc30ff1af80ea02a41fc404420aaa9b016662" Oct 30 20:38:29.376442 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:38:29.376403 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:38:31.515522 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:31.515492 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:38:31.696601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:31.696571 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:38:54.451283 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:54.451237 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:55.542987 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:38:55.542944 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:38:59.376799 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:38:59.376764 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:39:18.512153 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:18.512116 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:19.356996 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:19.356966 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:39:19.542003 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:19.541963 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:39:19.919135 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:19.919105 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:20.078350 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:20.078318 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:23.501625 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:23.501589 5733 scope.go:122] "RemoveContainer" containerID="549d744f38e13ca177c9055929eb71a1a3f8048e4b499f2112b82c6ceada484d" Oct 30 20:39:23.510616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:23.510596 5733 scope.go:122] "RemoveContainer" containerID="c10f3ae91ebf7dd4aa52364fee8aea4a816b13882315fe7e3abe5fb45c3762dd" Oct 30 20:39:23.654470 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:23.654438 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jllfr_56834a9b-e103-4f55-b54f-14e73b4fa1db/global-pull-secret-syncer/0.log" Oct 30 20:39:23.854718 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:23.854688 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" Oct 30 20:39:24.071588 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:24.071552 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-103.us-east-2.compute.internal_e4925bc4d897d213bab4d8b38932653e/haproxy/0.log" Oct 30 20:39:29.376700 ip-10-0-139-103 kubenswrapper[5733]: E1030 20:39:29.376657 5733 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nrp79" containerName="extractor" Oct 30 20:39:30.086199 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:30.086163 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5fc95fd96f-2wntz_0125b6d4-4e87-4bba-b004-48ad867d787d/authorino/0.log" Oct 30 20:39:30.175519 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:30.175488 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/0.log" Oct 30 20:39:30.175519 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:30.175514 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-2s5r5_56747b0a-051b-4e92-a51a-b71b394a2de2/manager/1.log" Oct 30 20:39:30.271599 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:30.271563 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-pcl5w_76e6166d-79fd-4d5c-adde-24ab3b2b0400/manager/0.log" Oct 30 20:39:31.933476 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:31.933447 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/alertmanager/0.log" Oct 30 20:39:31.979837 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:31.979807 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/config-reloader/0.log" Oct 30 20:39:32.018066 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.018026 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/kube-rbac-proxy-web/0.log" Oct 30 20:39:32.053584 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.053560 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/kube-rbac-proxy/0.log" Oct 30 20:39:32.090346 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.090311 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/kube-rbac-proxy-metric/0.log" Oct 30 20:39:32.126947 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.126925 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/prom-label-proxy/0.log" Oct 30 20:39:32.161388 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.161357 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ad8b1c40-5211-4b2d-82a2-e11a686f5f60/init-config-reloader/0.log" Oct 30 20:39:32.270506 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.270438 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-state-metrics/0.log" Oct 30 20:39:32.300592 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.300572 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-rbac-proxy-main/0.log" Oct 30 20:39:32.331204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.331179 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fx95g_d87433f0-7272-49cb-a84e-20565709bdae/kube-rbac-proxy-self/0.log" Oct 30 20:39:32.368310 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.368288 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd8695446-nbt67_38afb684-e999-4c10-9281-62f633431cc6/metrics-server/1.log" Oct 30 20:39:32.369510 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.369491 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bd8695446-nbt67_38afb684-e999-4c10-9281-62f633431cc6/metrics-server/0.log" Oct 30 20:39:32.742469 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.742438 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/node-exporter/0.log" Oct 30 20:39:32.788921 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.788895 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/kube-rbac-proxy/0.log" Oct 30 20:39:32.825567 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.825543 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tx4kh_a1c7a804-0740-4d51-884b-ea661daf80be/init-textfile/0.log" Oct 30 20:39:32.876146 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.876114 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-main/0.log" Oct 30 20:39:32.913164 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.913140 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/kube-rbac-proxy-self/0.log" Oct 30 20:39:32.945382 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.945349 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-6d6vf_e0d4ab99-d6b0-4b6c-99ed-ee24feac5df0/openshift-state-metrics/0.log" Oct 30 20:39:32.999203 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:32.999137 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/prometheus/0.log" Oct 30 20:39:33.019553 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.019522 5733 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs"] Oct 30 20:39:33.020020 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020002 5733 cpu_manager.go:414] "containerMap: removing container" podUID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" containerName="payload-pre-processing" Oct 30 20:39:33.020132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020022 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" containerName="payload-pre-processing" Oct 30 20:39:33.020132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020046 5733 cpu_manager.go:414] "containerMap: removing container" podUID="5d837f4e-68b7-4654-ab4e-8d16c253a358" containerName="payload-processing" Oct 30 20:39:33.020132 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020069 5733 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d837f4e-68b7-4654-ab4e-8d16c253a358" containerName="payload-processing" Oct 30 20:39:33.020304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020213 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed988cf9-4a6a-46ed-b0f7-dee6c9c40a6b" containerName="payload-pre-processing" Oct 30 20:39:33.020304 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.020232 5733 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d837f4e-68b7-4654-ab4e-8d16c253a358" containerName="payload-processing" Oct 30 20:39:33.024874 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.024855 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.035714 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.035686 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs"] Oct 30 20:39:33.036705 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.036687 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/config-reloader/0.log" Oct 30 20:39:33.041279 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.041250 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-lib-modules\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.041387 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.041309 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-sys\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.041479 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.041459 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw542\" (UniqueName: \"kubernetes.io/projected/80968cf5-8fba-442f-ae08-041097a1e2e8-kube-api-access-dw542\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.041536 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.041521 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-proc\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.041601 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.041588 5733 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-podres\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.071889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.071868 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/thanos-sidecar/0.log" Oct 30 20:39:33.110156 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.110134 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-web/0.log" Oct 30 20:39:33.112250 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.112234 5733 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rzbpr\"/\"default-dockercfg-czcz7\"" Oct 30 20:39:33.116107 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.116091 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"kube-root-ca.crt\"" Oct 30 20:39:33.116181 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.116150 5733 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rzbpr\"/\"openshift-service-ca.crt\"" Oct 30 20:39:33.142781 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.142746 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-podres\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.142894 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.142821 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-lib-modules\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.142894 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.142854 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-sys\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.142934 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw542\" (UniqueName: \"kubernetes.io/projected/80968cf5-8fba-442f-ae08-041097a1e2e8-kube-api-access-dw542\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143012 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.142952 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-podres\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.143014 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-lib-modules\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.143024 5733 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-proc\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143160 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.143098 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-proc\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143332 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.143216 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80968cf5-8fba-442f-ae08-041097a1e2e8-sys\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.143889 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.143872 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy/0.log" Oct 30 20:39:33.155347 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.155325 5733 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw542\" (UniqueName: \"kubernetes.io/projected/80968cf5-8fba-442f-ae08-041097a1e2e8-kube-api-access-dw542\") pod \"perf-node-gather-daemonset-hwzzs\" (UID: \"80968cf5-8fba-442f-ae08-041097a1e2e8\") " pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.176862 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.176836 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/kube-rbac-proxy-thanos/0.log" Oct 30 20:39:33.214916 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.214892 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ef38f10e-731b-49df-a5c8-0fce38cc59ab/init-config-reloader/0.log" Oct 30 20:39:33.339602 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.339527 5733 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:33.617394 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.617367 5733 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs"] Oct 30 20:39:33.619355 ip-10-0-139-103 kubenswrapper[5733]: W1030 20:39:33.619330 5733 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80968cf5_8fba_442f_ae08_041097a1e2e8.slice/crio-cce2476c06e0f353f0d57df8235cadebefd46383ef6b4e6f7670727c0f1696c6 WatchSource:0}: Error finding container cce2476c06e0f353f0d57df8235cadebefd46383ef6b4e6f7670727c0f1696c6: Status 404 returned error can't find the container with id cce2476c06e0f353f0d57df8235cadebefd46383ef6b4e6f7670727c0f1696c6 Oct 30 20:39:33.685396 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:33.685368 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" event={"ID":"80968cf5-8fba-442f-ae08-041097a1e2e8","Type":"ContainerStarted","Data":"cce2476c06e0f353f0d57df8235cadebefd46383ef6b4e6f7670727c0f1696c6"} Oct 30 20:39:34.692000 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:34.691966 5733 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" event={"ID":"80968cf5-8fba-442f-ae08-041097a1e2e8","Type":"ContainerStarted","Data":"978108cf3411a961f29c33b8ce098cf88d565db510384a5db7191a9bb1621d15"} Oct 30 20:39:34.692391 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:34.692091 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:34.711811 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:34.711764 5733 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" podStartSLOduration=2.711751569 podStartE2EDuration="2.711751569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:39:34.709685726 +0000 UTC m=+4134.777521294" watchObservedRunningTime="2026-10-30 20:39:34.711751569 +0000 UTC m=+4134.779587137" Oct 30 20:39:36.667741 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:36.667708 5733 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kms8d_46dad1ef-96d8-483e-909b-c082e47acbbd/konnectivity-agent/0.log" limit=1024 Oct 30 20:39:38.024525 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:38.024487 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/dns/1.log" Oct 30 20:39:38.025929 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:38.025909 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/dns/0.log" Oct 30 20:39:38.050190 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:38.050162 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bjt85_b107b211-0176-4170-8d0f-85d4b36a4f30/kube-rbac-proxy/0.log" Oct 30 20:39:38.313204 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:38.313123 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jcmd8_ce67c2e9-3fdd-4a2b-b627-70993c44ea9f/dns-node-resolver/0.log" Oct 30 20:39:39.100365 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:39.100332 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7gh2g_ffaa0f5e-c8f0-4e96-83a0-61a39edd65ce/node-ca/0.log" Oct 30 20:39:40.708452 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:40.708419 5733 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rzbpr/perf-node-gather-daemonset-hwzzs" Oct 30 20:39:43.359366 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.359336 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x4v2z_ff194676-b322-42b3-a970-7db28cc2ac3c/kube-auth-proxy/0.log" Oct 30 20:39:43.552613 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.552587 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/0.log" Oct 30 20:39:43.552774 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.552618 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/1.log" Oct 30 20:39:43.700507 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.700425 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ldpp9_393595d7-bbd7-4db8-b5a5-5db2cc93b72b/istio-proxy/0.log" Oct 30 20:39:43.753804 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.753776 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/1.log" Oct 30 20:39:43.753950 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.753853 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/0.log" Oct 30 20:39:43.928220 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:43.928193 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:39:46.579616 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.579580 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bct4d_0678842b-b564-4b68-ae5b-aaffd1680479/serve-healthcheck-canary/0.log" Oct 30 20:39:46.765471 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.765440 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d87896589-x4v2z_ff194676-b322-42b3-a970-7db28cc2ac3c/kube-auth-proxy/0.log" Oct 30 20:39:46.850281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.850238 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/0.log" Oct 30 20:39:46.850281 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.850261 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-8fcjn_f4dfd64f-0ef4-4176-9af6-a587e7e6c4cc/istio-proxy/1.log" Oct 30 20:39:46.923641 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.923610 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-ldpp9_393595d7-bbd7-4db8-b5a5-5db2cc93b72b/istio-proxy/0.log" Oct 30 20:39:46.953632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.948445 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/1.log" Oct 30 20:39:46.953632 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:46.948733 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-qr62s_32aa4d2b-e4fa-4a68-883e-082788e51c95/istio-proxy/0.log" Oct 30 20:39:47.045301 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:47.045266 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-6slx7_23ce4b26-7e58-442e-85bc-21bc1535c2fd/payload-processing/0.log" Oct 30 20:39:47.253742 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:47.253662 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/kube-rbac-proxy/0.log" Oct 30 20:39:47.284386 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:47.284361 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/exporter/0.log" Oct 30 20:39:47.316523 ip-10-0-139-103 kubenswrapper[5733]: I1030 20:39:47.316499 5733 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrp79_72fe1029-ecc8-4b99-9387-659ae6e414b3/extractor/0.log"